1

I have a Resque job which pulls a csv list of data off of a remote server and then runs through the +40k entries to add any new items to an existing database table. The job is running fine however it severely slows down the response time of any subsequent requests to the server. In the console which I've launched 'bundle exec rails server', I see not print statements though the job is running. However once I hit my rails server (via a page referesh), I see multiple SELECT / INSERT statements roll by before the server responds. The SELECT/INSERT statements are clearly generated by my Resque job but oddly they wait to print to the console unit I hit the server through the browser.

It sure feels like I'm doing something wrong or not following the 'rails way'. Advice?

Here is the code in my Resque job which does the SELECT/INSERTS

# data is an array of hashes formed from parsing csv input. Max size is 1000
ActiveRecord::Base.transaction do
  data.each do |h|
    MyModel.find_or_create_by_X_and_Y( h[:x], h[:y], h )
  end
end

Software Stack

  • Rails 3.2.0
  • postgresql 9.1
  • Resque 1.20.0

EDIT

I've finally take the time to debug this a bit more. Even a very simple worker, like below, slows down the next server response. In the console where I've launched the rail sever process I see that the delay occurs b/c stdout from the worker is being printed only after I ping the server.

  def perform()
    s = Time.now
    0.upto( 90000 ) do |i|
      Rails.logger.debug  i * i
    end
    e = Time.now
    Rails.logger.info "Start: #{s} ---- End #{e}"
    Rails.logger.info "Total Time: #{e - s }"
  end

I can get the rails server back to its normal responsiveness again if I suppress stdout when I launch rails but it doesn't seem like that should be necessary... bundle exec rails server > /dev/nul

Any input on a better way to solve this issue?

4

1 に答える 1

0

「Resqueのロギングの問題」に対するこの回答が役立つと思います。

開発モードの Rails サーバーでは、ログ ファイルが開かれています。私の理解では、これを確認する必要がありますが、順序を維持するために、新しいものを書き込む前にログをフラッシュするということです。Rails サーバーを端末に接続している場合、最初にすべての変更を出力する必要があります。これにより、ワーカーがログに大量の書き込みを行った場合、大幅な遅延が発生する可能性があります。

注：これはしばらくの間私に起こっていましたが、最近指を置いただけです。

于 2012-08-14T23:35:16.927 に答える