0

生産エラーが発生しました。誰かが教えてくれることを願っています。AWS のクラスター (2 dbs + アービター) で MongoDB AMI インスタンス (1000 IOPS) を実行しています。データベースに接続するアプリ サーバーは、別のインスタンスに存在します。

何週間も正常に実行した後、以下の例外が発生しました。これにより、アプリサーバーノードを再起動するまで、db アクセスが効果的にシャットダウンされました。システムのトラフィックは非常に少ないです。つまり、サーバーが非難されているようには見えません (1 桁の CPU 使用率)。

以下は私のmongo構成オプション設定です:

auto_connect_retry = false
connections_per_host = 100
threads_multiplier = 50
max_wait_time = 120000
connect_timeout = 10000
socket_timeout = 60000

何が起こったのかを理解し、次にそれを防ぐために何かできるかどうかを理解する必要があります。

18:14:15.482 AWECluster-akka.actor.default-dispatcher-3 ERROR akka.actor.OneForOneStrategy - Write operation to server /10.0.8.10:27017 failed on database awe_prod_preview
com.mongodb.MongoException$Network: Write operation to server /10.0.8.10:27017 failed on database awe_prod_preview
    at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:153) ~[deps.jar:0.1-SNAPSHOT]
    at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:115) ~[deps.jar:0.1-SNAPSHOT]
    at com.mongodb.DBApiLayer$MyCollection.update(DBApiLayer.java:327) ~[deps.jar:0.1-SNAPSHOT]
    at com.mongodb.DBCollection.update(DBCollection.java:178) ~[deps.jar:0.1-SNAPSHOT]
    at com.mongodb.DBCollection.save(DBCollection.java:818) ~[deps.jar:0.1-SNAPSHOT]
    at com.mongodb.casbah.MongoCollectionBase$class.save(MongoCollection.scala:573) ~[deps.jar:0.1-SNAPSHOT]
    at com.mongodb.casbah.MongoCollection.save(MongoCollection.scala:866) ~[deps.jar:0.1-SNAPSHOT]
    at com.novus.salat.dao.SalatDAO.save(SalatDAO.scala:404) ~[deps.jar:0.1-SNAPSHOT]
    at com.novus.salat.dao.ModelCompanion$class.save(ModelCompanion.scala:272) ~[deps.jar:0.1-SNAPSHOT]
    at awe.etl._ETL$$anonfun$5$$anon$2.save(ETL.scala:75) ~[shock-etl.jar:0.3.22]
    at com.novus.salat.dao.BaseDAOMethods$class.save(DAO.scala:127) ~[deps.jar:0.1-SNAPSHOT]
    at awe.etl._ETL$$anonfun$5$$anon$2.save(ETL.scala:75) ~[shock-etl.jar:0.3.22]
    at awe.etl.step.Publisher.publishNow(Publish.scala:24) ~[shock-etl.jar:0.3.22]
    at awe.etl.ETLActor$$anonfun$receive$1$$anonfun$applyOrElse$4$$anonfun$apply$4.apply(ETLActor.scala:116) ~[shock-etl.jar:0.3.22]
    at awe.etl.ETLActor$$anonfun$receive$1$$anonfun$applyOrElse$4$$anonfun$apply$4.apply(ETLActor.scala:114) ~[shock-etl.jar:0.3.22]
    at scala.Option.fold(Option.scala:157) ~[deps.jar:0.1-SNAPSHOT]
    at awe.etl.ETLActor$$anonfun$receive$1$$anonfun$applyOrElse$4.apply(ETLActor.scala:114) ~[shock-etl.jar:0.3.22]
    at awe.etl.ETLActor$$anonfun$receive$1$$anonfun$applyOrElse$4.apply(ETLActor.scala:113) ~[shock-etl.jar:0.3.22]
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) ~[deps.jar:0.1-SNAPSHOT]
    at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108) ~[deps.jar:0.1-SNAPSHOT]
    at awe.etl.ETLActor$$anonfun$receive$1.applyOrElse(ETLActor.scala:113) ~[shock-etl.jar:0.3.22]
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) [deps.jar:0.1-SNAPSHOT]
    at akka.actor.ActorCell.invoke(ActorCell.scala:456) [deps.jar:0.1-SNAPSHOT]
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) [deps.jar:0.1-SNAPSHOT]
    at akka.dispatch.Mailbox.run(Mailbox.scala:219) [deps.jar:0.1-SNAPSHOT]
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) [deps.jar:0.1-SNAPSHOT]
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [deps.jar:0.1-SNAPSHOT]
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [deps.jar:0.1-SNAPSHOT]
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [deps.jar:0.1-SNAPSHOT]
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [deps.jar:0.1-SNAPSHOT]
Caused by: java.net.SocketException: Connection timed out
    at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.6.0_24]
    at java.net.SocketInputStream.read(SocketInputStream.java:146) ~[na:1.6.0_24]
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) ~[na:1.6.0_24]
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:275) ~[na:1.6.0_24]
    at java.io.BufferedInputStream.read(BufferedInputStream.java:334) ~[na:1.6.0_24]
    at org.bson.io.Bits.readFully(Bits.java:46) ~[deps.jar:0.1-SNAPSHOT]
    at org.bson.io.Bits.readFully(Bits.java:33) ~[deps.jar:0.1-SNAPSHOT]
    at org.bson.io.Bits.readFully(Bits.java:28) ~[deps.jar:0.1-SNAPSHOT]
    at com.mongodb.Response.<init>(Response.java:40) ~[deps.jar:0.1-SNAPSHOT]
    at com.mongodb.DBPort.go(DBPort.java:142) ~[deps.jar:0.1-SNAPSHOT]
    at com.mongodb.DBPort.go(DBPort.java:106) ~[deps.jar:0.1-SNAPSHOT]
    at com.mongodb.DBPort.findOne(DBPort.java:162) ~[deps.jar:0.1-SNAPSHOT]
    at com.mongodb.DBPort.runCommand(DBPort.java:170) ~[deps.jar:0.1-SNAPSHOT]
    at com.mongodb.DBTCPConnector._checkWriteError(DBTCPConnector.java:100) ~[deps.jar:0.1-SNAPSHOT]
    at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:142) ~[deps.jar:0.1-SNAPSHOT]
    ... 29 common frames omitted
4

1 に答える 1