私の map-reduce ジョブでは、reducer の数を 0 に設定すると、プログラムは 5 分未満で正常に終了します。ただし、レデューサーの数を 0 ではなく (1 から 40 の間) に設定し、レデューサー クラスを指定していない場合でも、多くの失敗したタスクで処理が非常に遅くなり、次のエラー メッセージが表示されて突然失敗します。レデューサー クラスを指定していないため、このエラーの唯一の原因はマッパーの出力の並べ替えです。マッパーは、約 27,000 の異なるキーを生成します。ここで何が問題なのですか？ログファイルにも役立つものはありません。私が持っている唯一の推測は、多数のキーがこの問題を引き起こすということです。どうすればこの問題を解決できますか?
13/10/28 13:06:40 INFO mapred.JobClient: map 70% reduce 18%
13/10/28 13:07:13 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000052_2, Status : FAILED
Task attempt_201310101346_0661_m_000052_2 failed to report status for 600 seconds. Killing!
13/10/28 13:07:19 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000059_2, Status : FAILED
Task attempt_201310101346_0661_m_000059_2 failed to report status for 600 seconds. Killing!
13/10/28 13:07:23 INFO mapred.JobClient: map 70% reduce 19%
13/10/28 13:07:24 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000060_2, Status : FAILED
Task attempt_201310101346_0661_m_000060_2 failed to report status for 600 seconds. Killing!
13/10/28 13:07:26 INFO mapred.JobClient: map 70% reduce 20%
13/10/28 13:07:47 INFO mapred.JobClient: map 71% reduce 20%
13/10/28 13:07:54 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000061_2, Status : FAILED
Task attempt_201310101346_0661_m_000061_2 failed to report status for 600 seconds. Killing!
13/10/28 13:08:22 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000062_2, Status : FAILED
Task attempt_201310101346_0661_m_000062_2 failed to report status for 600 seconds. Killing!
13/10/28 13:09:03 INFO mapred.JobClient: map 72% reduce 20%
13/10/28 13:09:15 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000063_2, Status : FAILED
Task attempt_201310101346_0661_m_000063_2 failed to report status for 600 seconds. Killing!
13/10/28 13:09:18 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000066_2, Status : FAILED
Task attempt_201310101346_0661_m_000066_2 failed to report status for 600 seconds. Killing!
13/10/28 13:09:20 INFO mapred.JobClient: Task Id : attempt_201310101346_0661_m_000068_2, Status : FAILED
Task attempt_201310101346_0661_m_000068_2 failed to report status for 600 seconds. Killing!
13/10/28 13:09:55 INFO mapred.JobClient: map 72% reduce 21%
13/10/28 13:10:30 INFO mapred.JobClient: map 72% reduce 22%
13/10/28 13:10:33 INFO mapred.JobClient: map 73% reduce 22%
13/10/28 13:12:21 INFO mapred.JobClient: map 74% reduce 22%
13/10/28 13:13:40 INFO mapred.JobClient: map 74% reduce 23%
13/10/28 13:14:09 INFO mapred.JobClient: Job complete: job_201310101346_0661
13/10/28 13:14:09 INFO mapred.JobClient: Counters: 31
13/10/28 13:14:09 INFO mapred.JobClient: File System Counters
13/10/28 13:14:09 INFO mapred.JobClient: FILE: Number of bytes read=0
13/10/28 13:14:09 INFO mapred.JobClient: FILE: Number of bytes written=542433323
13/10/28 13:14:09 INFO mapred.JobClient: FILE: Number of read operations=0
13/10/28 13:14:09 INFO mapred.JobClient: FILE: Number of large read operations=0
13/10/28 13:14:09 INFO mapred.JobClient: FILE: Number of write operations=0
13/10/28 13:14:09 INFO mapred.JobClient: HDFS: Number of bytes read=316452240
13/10/28 13:14:09 INFO mapred.JobClient: HDFS: Number of bytes written=0
13/10/28 13:14:09 INFO mapred.JobClient: HDFS: Number of read operations=100
13/10/28 13:14:09 INFO mapred.JobClient: HDFS: Number of large read operations=0
13/10/28 13:14:09 INFO mapred.JobClient: HDFS: Number of write operations=0
13/10/28 13:14:09 INFO mapred.JobClient: Job Counters
13/10/28 13:14:09 INFO mapred.JobClient: Failed map tasks=1
13/10/28 13:14:09 INFO mapred.JobClient: Launched map tasks=169
13/10/28 13:14:09 INFO mapred.JobClient: Launched reduce tasks=1
13/10/28 13:14:09 INFO mapred.JobClient: Data-local map tasks=23
13/10/28 13:14:09 INFO mapred.JobClient: Rack-local map tasks=146
13/10/28 13:14:09 INFO mapred.JobClient: Total time spent by all maps in occupied slots (ms)=72419457
13/10/28 13:14:09 INFO mapred.JobClient: Total time spent by all reduces in occupied slots (ms)=1194624
13/10/28 13:14:09 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
13/10/28 13:14:09 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
13/10/28 13:14:09 INFO mapred.JobClient: Map-Reduce Framework
13/10/28 13:14:09 INFO mapred.JobClient: Map input records=6894035
13/10/28 13:14:09 INFO mapred.JobClient: Map output records=6422195
13/10/28 13:14:09 INFO mapred.JobClient: Map output bytes=2300749768
13/10/28 13:14:09 INFO mapred.JobClient: Input split bytes=6200
13/10/28 13:14:09 INFO mapred.JobClient: Combine input records=6422195
13/10/28 13:14:09 INFO mapred.JobClient: Combine output records=801608
13/10/28 13:14:09 INFO mapred.JobClient: Spilled Records=801608
13/10/28 13:14:09 INFO mapred.JobClient: CPU time spent (ms)=48077980
13/10/28 13:14:09 INFO mapred.JobClient: Physical memory (bytes) snapshot=13246660608
13/10/28 13:14:09 INFO mapred.JobClient: Virtual memory (bytes) snapshot=20080971776
13/10/28 13:14:09 INFO mapred.JobClient: Total committed heap usage (bytes)=10181214208
13/10/28 13:14:09 INFO mapred.JobClient: org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
13/10/28 13:14:09 INFO mapred.JobClient: BYTES_READ=316416597
13/10/28 13:14:09 INFO mapred.JobClient: Job Failed: NA
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1373)
at Mondrian8.main(Mondrian8.java:547)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)