clojure - How to use run distribute tasks on worker nodes in a Clojure app?

Question

On Python/Django stack, we were used to using Celery along with RabbitMQ. Everything was easily done.

However when we tried doing the same thing in Clojure land, what we could get was Langhour. In our current naive implementation we have a worker system which has three core parts.

• Publisher module
• Subscriber module
• Task module

We can start the system on any node in either publisher or subscriber mode. They are connected to RabbitMQ server.

They share one worker_queue.

What we are doing is creating tasks in Task module, and then when we want to run a task on subscriber. we send an expression call to the method, in EDN format to Subscriber which then decodes this and runs the actual task using eval.

Now is using eval safe ? we are not running expressions generated by user or any third party system.Initially we were planning to use JSON to send the payload message but then EDN gave us a lot more flexibility and it works like a charm, as of now.

Also is there a better way to do this ?

Answer 1

ニーズ (およびチーム) によって異なりますが、 Storm Projectを強くお勧めします。分散型でフォールト トレラントなリアルタイム計算が得られ、非常に使いやすいです。

Storm のもう 1 つの優れた点は、トポロジのデータソースとして多数のオプションがサポートされていることです。たとえば、Apache Kafka、RabbitMQ、Kestrel、MongoDB などです。満足できない場合は、独自のドライバーを作成できます。

また、トポロジで何が起こっているかを確認するための Web インターフェイスも備えています。