perl - A queueing system for Perl

Question

I'm working on a Perl project which needs a FIFO message queue for distributing tasks between several processes on a single machine (UNIX). The queue size may grow up to 1M jobs.

I've tried IPC::DirQueue, but it becomes awfully slow with 50k or so jobs enqueued. What are good alternatives to this module which can be used in Perl?

Answer 1

I've had pretty good success with using ZeroMQ for this sort of problem, both with Perl and other languages.

In my experience, the ZeroMQ module appears to be the most reliable binding for Perl currently.

Answer 2

私はあなたがリストした条件の下でそれを試していませんが、Thread::Queueは私にとって有用であることが証明されています。フォークと組み合わせると、プロセスがキュー作成者によって生成されたものである限り、プロセスとの通信に使用できます。

use forks;  # If you want to use processes instead of threads.
use Thread::Queue qw( );

通常、ワーカーモデルが理想的です。

my $q = Thread::Queue->new();

my @workers;
for (1..$NUM_WORKERS) {
   push @workers, async {
      while (my $item = $q->dequeue()) {
         ...
      }
   };
}

# ... Enqueue requests [ $q->enqueue($request); ] ...

# Signal termination
$q->enqueue(undef) for 1..@workers;

# Collect workers.
$_->join() for @workers;