2

I have several servers that produce xml files from Python and some other servers that consume those xmls using Java. I've only recently looked into JMS and ActiveMQ and decided to try using it to pass the xml files.

So I set up ActiveMQ daemons on the consumer servers and figured I'd implement some cycling method on the produces to distribute the xmls evenly across the consumers.

Python  -----------------------------> ActiveMQ ---> Java
        \                           /
         \                         /
          ---------------------------> ActiveMQ ---> Java
                                 /  /
Python  ----------------------------

For testing, I ran one producer and one consumer and looked at the results.

To my surprise, the messages from the producer were distributed across all the ActiveMQ servers on the network. Since I only ran one consumer it only received the xmls that got to the ActiveMQ daemon on that machine, and the rest of the xmls were waiting patiently on the other ActiveMQ daemons on other machines.

Python  -----------------------------> ActiveMQ ---> Java (work)
                                          |
                                          |
                                       ActiveMQ (xmls piling up)

EDIT: This is not what actually happened, sorry. See below for details

Now, I'm not complaining, since this is what I wanted anyway, but I'm a little confused: what is the proper way to implement this many-to-many queue that I'm after?

Should I set up ActiveMQ daemons on my producer machines as well, send the xmls to the localhost ActiveMQs and trust the automatic discovery to get the xmls to consumers?

Python ---> ActiveMQ ------------------------------ ActiveMQ ---> Java
                |                                      |
                |                                      |
                |                                -- ActiveMQ ---> Java
                |                               |  
Python ---> ActiveMQ----------------------------

Should I stick to my original plan, and cycle the messages across the consumer machines, just to be safe?

Or is there an API I should use that hide those details from my processes?

BTW, the producers are python processes using STOMP and the consumers are java using JMS.

I apologize if your eyes are hurting from my crappy ASCII art, I wasn't sure if I'm being clear enough with just words.

EDIT

Apparently, when I was running "one producer and one consumer" I didn't notice that the other consumers were already running. They were just not doing anything useful with the xmls they processed. That's why I was seeing partial results.

After reading a bit more and experimenting a little, I figured out the following:

By default, ActiveMQ will auto-discover other ActiveMQ instances on the local network, and create a store-and-forward network of brokers. This means that the producer can post the xmls to any ActiveMQ instance and they will find their way to consumers listening on other ActiveMQ instances on the same network.

Note that the documentation claims that auto-discovery is not recommended for production setups.

The accepted answer below still holds true, though. The simplest way to use ActiveMQ is just use one or two servers as "Queue Servers". Nevertheless, I chose to go with my original plan, because I think it will reduce network traffic (with a middle server, xmls have to go into it and the out of it again).

4

1 に答える 1

1

イサドク、あなたはおそらくメッセージングの使用を正しく考えていないと思います.

消費者ごとに 1 つのケースで MOM インスタンス (ActiveMQ、RabbitMQ、またはその他の MOM ブローカー) を持つことは、概念的にはあまり意味がありません。むしろ、MOM ブローカーをメッセージのルーターと考えるのが最善です。

その場合、すべてのプロデューサーとすべてのコンシューマーが接続する 1 つの ActiveMQ ブローカー インスタンス (スケーリングの問題がある場合はシャーディングまたはスケーリングされるか、HA の考慮事項がある場合はレプリケートされる可能性があります) があります。次に、すべての XML が同じブローカー インスタンスに送られ、すべてのコンシューマーが同じブローカー インスタンスから読み取ります。その場合、ブローカは、使用するヒューリスティックに基づいて、メッセージが送信されるコンシューマを決定します。

これはまた、プロデューサーとコンシューマーを動的に追加および削除でき、何も変更されないことを意味します。それらはすべて同じブローカーに接続するため、負荷が変化したり、システムに障害が発生したりしたときに、プロデューサーとコンシューマーを追加および削除できます。

于 2009-03-23T14:13:20.527 に答える