0

My application listen on kafka topic and dump data into cassandra. Threads loads some information from mongo too. Lag in kafka topic getting increased. I have seen that mostly threads are blocked while loading some class. I am attaching my thread_dump below.

 "KafkaConsumer-49" prio=10 tid=0x00007f1178fdd000 nid=0x78e0 waiting for monitor entry [0x00007f1155fb5000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:403)
        - waiting to lock <0x00000006c0655b58> (a java.lang.Object)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
        at org.springframework.util.ClassUtils.forName(ClassUtils.java:258)
        at org.springframework.data.convert.SimpleTypeInformationMapper.resolveTypeFrom(SimpleTypeInformationMapper.java:56)
        at org.springframework.data.convert.DefaultTypeMapper.readType(DefaultTypeMapper.java:103)
        at org.springframework.data.convert.DefaultTypeMapper.getDefaultedTypeToBeUsed(DefaultTypeMapper.java:144)
        at org.springframework.data.convert.DefaultTypeMapper.readType(DefaultTypeMapper.java:121)
        at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:186)
        at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:176)
        at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:172)
        at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:75)
        at org.springframework.data.mongodb.core.MongoTemplate$ReadDbObjectCallback.doWith(MongoTemplate.java:1840)
        at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:1536)
        at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1336)
        at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1322)
        at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:495)
        at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:486)
        at com.snapdeal.coms.timemachine.mao.TimeMachineMao.getVendorProductsForUploadId(TimeMachineMao.java:32)
        at com.snapdeal.coms.timemachine.service.TimeMachineService.getVendorProductsForUploadIdAndSupc(TimeMachineService.java:35)
        at com.snapdeal.coms.timemachine.event.SupcUploadIdStateUpdateEventHandler.handleEvent(SupcUploadIdStateUpdateEventHandler.java:40)



KafkaConsumer-48" prio=10 tid=0x00007f1178fdb000 nid=0x78df waiting for monitor entry [0x00007f11560b6000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:403)
        - waiting to lock <0x00000006c0655b58> (a java.lang.Object)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
        at org.springframework.util.ClassUtils.forName(ClassUtils.java:258)
        at org.springframework.data.convert.SimpleTypeInformationMapper.resolveTypeFrom(SimpleTypeInformationMapper.java:56)
        at org.springframework.data.convert.DefaultTypeMapper.readType(DefaultTypeMapper.java:103)
        at org.springframework.data.convert.DefaultTypeMapper.getDefaultedTypeToBeUsed(DefaultTypeMapper.java:144)
        at org.springframework.data.convert.DefaultTypeMapper.readType(DefaultTypeMapper.java:121)
        at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:186)
        at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:176)
        at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:172)
        at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:75)
        at org.springframework.data.mongodb.core.MongoTemplate$ReadDbObjectCallback.doWith(MongoTemplate.java:1840)
        at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:1536)
        at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1336)
        at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1322)
        at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:495)
        at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:486)
        at com.snapdeal.coms.timemachine.mao.TimeMachineMao.getVendorProductsForUploadId(TimeMachineMao.java:32)
        at com.snapdeal.coms.timemachine.service.TimeMachineService.getVendorProductsForUploadIdAndSupc(TimeMachineService.java:35)
        at com.snapdeal.coms.timemachine.event.SupcUploadIdStateUpdateEventHandler.handleEvent(SupcUploadIdStateUpdateEventHandler.java:40)
        at com.snapdeal.coms.timemachine.TimeMachine.onEvent(TimeMachine.java:109)

"KafkaConsumer-47" prio=10 tid=0x00007f1178fd9800 nid=0x78de waiting for monitor entry [0x00007f11561b7000] java.lang.Thread.State: BLOCKED (on object monitor) at java.lang.ClassLoader.loadClass(ClassLoader.java:403) - waiting to lock <0x00000006c0655b58> (a java.lang.Object) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:356) at org.springframework.util.ClassUtils.forName(ClassUtils.java:258) at org.springframework.data.convert.SimpleTypeInformationMapper.resolveTypeFrom(SimpleTypeInformationMapper.java:56) at org.springframework.data.convert.DefaultTypeMapper.readType(DefaultTypeMapper.java:103) at org.springframework.data.convert.DefaultTypeMapper.getDefaultedTypeToBeUsed(DefaultTypeMapper.java:144) at org.springframework.data.convert.DefaultTypeMapper.readType(DefaultTypeMapper.java:121) at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:186) at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:176) at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:172) at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:75) at org.springframework.data.mongodb.core.MongoTemplate$ReadDbObjectCallback.doWith(MongoTemplate.java:1840) at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:1536) at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1336) at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1322) at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:495) at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:486)

"KafkaConsumer-46" prio=10 tid=0x00007f1178fd8000 nid=0x78dd waiting for monitor entry [0x00007f11562b8000]

java.lang.Thread.State: BLOCKED (on object monitor) at java.lang.ClassLoader.loadClass(ClassLoader.java:403) - waiting to lock <0x00000006c0655b58> (a java.lang.Object) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:356) at org.springframework.util.ClassUtils.forName(ClassUtils.java:258) at org.springframework.data.convert.SimpleTypeInformationMapper.resolveTypeFrom(SimpleTypeInformationMapper.java:56) at org.springframework.data.convert.DefaultTypeMapper.readType(DefaultTypeMapper.java:103) at org.springframework.data.convert.DefaultTypeMapper.getDefaultedTypeToBeUsed(DefaultTypeMapper.java:144) at org.springframework.data.convert.DefaultTypeMapper.readType(DefaultTypeMapper.java:121) at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:186) at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:176) at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:172) at org.springframework.data.mongodb.core.convert.MappingMongoConverter.read(MappingMongoConverter.java:75) at org.springframework.data.mongodb.core.MongoTemplate$ReadDbObjectCallback.doWith(MongoTemplate.java:1840) at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:1536) at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1336) at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1322) at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:495) at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:486) at com.snapdeal.coms.timemachine.mao.TimeMachineMao.getVendorProductsForUploadId(TimeMachineMao.java:32) at com.snapdeal.coms.timemachine.service.TimeMachineService.getVendorProductsForUploadIdAndSupc(TimeMachineService.java:35) at com.snapdeal.coms.timemachine.event.SupcUploadIdStateUpdateEventHandler.handleEvent(SupcUploadIdStateUpdateEventHandler.java:40)

I am not sure why all the threads are blocked. I thought class get loaded only one time and later no need to take any lock .

4

2 に答える 2

0

を使用してConsumerOffsetChecker、消費者がまだ生きているかどうかを確認しましたか? $KAFKA_ROOT_DIR/フォルダー内から次のコマンドを試すことができます

  bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group consumer-group1 --zkconnect zkhost:zkport --topic topic1

FAQページから取ったいくつかのメモを次に示します

コンシューマー オフセットがしばらく経っても動かない場合は、コンシューマーが停止している可能性があります。コンシューマー オフセットが移動しているが、コンシューマー ラグ (ログの最後とコンシューマー オフセットの差) が増加している場合、コンシューマーはプロデューサーよりも低速です。コンシューマが遅い場合の一般的な解決策は、コンシューマの並列度を高めることです。これには、トピックのパーティション数を増やす必要がある場合があります。

上記の FAQ ページでは、コンシューマーがブロックされる理由についても説明しています。

于 2014-04-01T17:04:49.543 に答える
0

問題は、mongo からのデータの取得にありました。膨大なデータがあり、ページネーションが実装されておらず、特定のリクエストにソケット タイムアウトがなかったため、スレッドがブロックされていました。

于 2014-06-11T19:15:58.743 に答える