1

約 500 ノードのクラスターで Hadoop バージョン 1.0.0 を実行しています。私の仕事には、約 3000 の map タスクと 10 の reduce タスクがあります。マップ タスクは、約 4 時間後に完了します (予想どおり)。reduce タスクはそれぞれすぐに完了し、結果はすべて出力ディレクトリで利用できます。ただし、ジョブトラッカーは、一部のマップ タスクが失敗したと判断し、再実行を開始します。実行中および保留中の削減タスクの数はゼロのままです。最終的に約 8 時間後、これらのマップ タスクの最後のタスクが最終的に正常に完了し、ジョブは正常に完了したとマークされます。

何か案は？？？

以下は、jobtracker ログ ファイルの一部の抜粋です。

// map tasks all complete, eg:
2013-05-20 10:50:59,742 INFO org.apache.hadoop.mapred.JobInProgress:   
Task 'attempt_201305131710_0007_m_000430_0' has completed task_201305131710_0007_m_000430 successfully.

//reduce tasks all complete:

2013-05-20 13:38:34,040 INFO org.apache.hadoop.mapred.JobInProgress:         Task 'attempt_201305131710_0007_r_000009_0' has completed task_201305131710_0007_r_000009 successfully.
2013-05-20 13:38:34,142 INFO org.apache.hadoop.mapred.JobInProgress:    
Task 'attempt_201305131710_0007_r_000004_0' has completed task_201305131710_0007_r_000004 successfully.
2013-05-20 13:38:34,204 INFO org.apache.hadoop.mapred.JobInProgress:    
Task 'attempt_201305131710_0007_r_000008_0' has completed task_201305131710_0007_r_000008 successfully.
2013-05-20 13:38:34,745 INFO org.apache.hadoop.mapred.JobInProgress:   
Task 'attempt_201305131710_0007_r_000002_0' has completed task_201305131710_0007_r_000002 successfully.
2013-05-20 13:38:35,521 INFO org.apache.hadoop.mapred.JobInProgress:    
Task 'attempt_201305131710_0007_r_000003_0' has completed task_201305131710_0007_r_000003 successfully.
2013-05-20 13:38:36,196 INFO org.apache.hadoop.mapred.JobInProgress:   
Task 'attempt_201305131710_0007_r_000007_0' has completed task_201305131710_0007_r_000007 successfully.
2013-05-20 13:38:36,276 INFO org.apache.hadoop.mapred.JobTracker: Adding tracker tracker_HN301-1657.labs.edu.au:127.0.0.1/127.0.0.1:1295 to host HN301-1657.labs.edu.au
2013-05-20 13:38:36,469 INFO org.apache.hadoop.mapred.JobInProgress:   
Task 'attempt_201305131710_0007_r_000005_0' has completed task_201305131710_0007_r_000005 successfully.
2013-05-20 13:38:36,598 INFO org.apache.hadoop.mapred.JobInProgress:  
Task 'attempt_201305131710_0007_r_000006_0' has completed task_201305131710_0007_r_000006 successfully.
2013-05-20 13:38:36,612 INFO org.apache.hadoop.mapred.JobInProgress:   
Task 'attempt_201305131710_0007_r_000000_0' has completed task_201305131710_0007_r_000000 successfully.
2013-05-20 13:38:40,388 INFO org.apache.hadoop.mapred.JobInProgress:  
Task 'attempt_201305131710_0007_r_000001_0' has completed task_201305131710_0007_r_000001 successfully.
2013-05-20 13:44:12,795 INFO org.apache.hadoop.mapred.JobTracker: Lost tracker 'tracker_HN301-1657.labs.edu.au:127.0.0.1/127.0.0.1:3896'

//As the reduce tasks are reporting success, the job tracker detects that one of the job trackers has died and so restarts it.
//Each of the jobs previously completed successfully by that task tracker are then reexecuted
2013-05-20 13:44:12,795 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201305131710_0007_m_000430_0: Lost task tracker: tracker_HN301-1657.labs.edu.au:127.0.0.1/127.0.0.1:3896
2013-05-20 13:44:12,795 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201305131710_0007_m_000571_0: Lost task tracker: tracker_HN301-1657.labs.edu.au:127.0.0.1/127.0.0.1:3896
2013-05-20 13:44:12,796 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201305131710_0007_m_001612_0: Lost task tracker: tracker_HN301-1657.labs.edu.au:127.0.0.1/127.0.0.1:3896
2013-05-20 13:44:12,796 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201305131710_0007_m_001629_0: Lost task tracker: tracker_HN301-1657.labs.edu.au:127.0.0.1/127.0.0.1:3896
2013-05-20 13:44:12,796 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201305131710_0007_m_001892_0: Lost task tracker: tracker_HN301-1657.labs.edu.au:127.0.0.1/127.0.0.1:3896
2013-05-20 13:44:12,796 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201305131710_0007_m_002424_0: Lost task tracker: tracker_HN301-1657.labs.edu.au:127.0.0.1/127.0.0.1:3896
2013-05-20 13:44:12,796 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201305131710_0007_m_002437_0: Lost task tracker: tracker_HN301-1657.labs.edu.au:127.0.0.1/127.0.0.1:3896
2013-05-20 13:44:12,796 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201305131710_0007_m_002696_0: Lost task tracker: tracker_HN301-1657.labs.edu.au:127.0.0.1/127.0.0.1:3896
2013-05-20 13:44:12,796 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201305131710_0007_m_003130_0: Lost task tracker: tracker_HN301-1657.labs.edu.au:127.0.0.1/127.0.0.1:3896
2013-05-20 13:44:12,796 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201305131710_0007_m_003149_0: Lost task tracker: tracker_HN301-1657.labs.edu.au:127.0.0.1/127.0.0.1:3896
2013-05-20 13:44:12,796 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201305131710_0007_m_003187_0: Lost task tracker: tracker_HN301-1657.labs.edu.au:127.0.0.1/127.0.0.1:3896
2013-05-20 13:44:12,797 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201305131710_0007_m_003275_0: Lost task tracker: tracker_HN301-1657.labs.edu.au:127.0.0.1/127.0.0.1:3896
2013-05-20 13:44:12,797 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201305131710_0007_m_003358_0: Lost task tracker: tracker_HN301-1657.labs.edu.au:127.0.0.1/127.0.0.1:3896
2013-05-20 13:44:12,797 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201305131710_0007_m_003437_0: Lost task tracker: tracker_HN301-1657.labs.edu.au:127.0.0.1/127.0.0.1:3896
2013-05-20 13:44:12,797 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201305131710_0007_m_003451_0: Lost task tracker: tracker_HN301-1657.labs.edu.au:127.0.0.1/127.0.0.1:3896
2013-05-20 13:44:12,797 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201305131710_0007_m_003478_0: Lost task tracker: tracker_HN301-1657.labs.edu.au:127.0.0.1/127.0.0.1:3896
2013-05-20 13:44:12,797 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201305131710_0007_m_003506_0: Lost task tracker: tracker_HN301-1657.labs.edu.au:127.0.0.1/127.0.0.1:3896
2013-05-20 13:44:12,797 INFO org.apache.hadoop.mapred.TaskInProgress: Error from attempt_201305131710_0010_m_000021_0: Lost task tracker: tracker_HN301-1657.labs.edu.au:127.0.0.1/127.0.0.1:3896
2013-05-20 13:44:12,797 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201305131710_0010_m_000021_0'
2013-05-20 13:44:12,797 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201305131710_0007_m_000430_0'
2013-05-20 13:44:12,797 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201305131710_0007_m_000571_0'
2013-05-20 13:44:12,797 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201305131710_0007_m_001612_0'
2013-05-20 13:44:12,797 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201305131710_0007_m_001629_0'
2013-05-20 13:44:12,797 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201305131710_0007_m_001892_0'
2013-05-20 13:44:12,797 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201305131710_0007_m_002424_0'
2013-05-20 13:44:12,797 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201305131710_0007_m_002437_0'
2013-05-20 13:44:12,797 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201305131710_0007_m_002696_0'
2013-05-20 13:44:12,797 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201305131710_0007_m_003130_0'
2013-05-20 13:44:12,797 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201305131710_0007_m_003149_0'
2013-05-20 13:44:12,797 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201305131710_0007_m_003187_0'
2013-05-20 13:44:12,797 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201305131710_0007_m_003275_0'
2013-05-20 13:44:12,797 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201305131710_0007_m_003358_0'
2013-05-20 13:44:12,797 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201305131710_0007_m_003437_0'
2013-05-20 13:44:12,797 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201305131710_0007_m_003451_0'
2013-05-20 13:44:12,797 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201305131710_0007_m_003478_0'
2013-05-20 13:44:12,797 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201305131710_0007_m_003506_0'
2013-05-20 13:44:12,917 INFO org.apache.hadoop.mapred.JobTracker: Adding task (TASK_CLEANUP) 'attempt_201305131710_0010_m_000021_0' to tip task_201305131710_0010_m_000021, for   tracker 'tracker_HN301-1654.labs.edu.au:127.0.0.1/127.0.0.1:1100'
2013-05-20 13:44:13,760 INFO org.apache.hadoop.mapred.JobInProgress: Choosing a failed task task_201305131710_0007_m_000430
2013-05-20 13:44:13,761 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201305131710_0007_m_000430_1' to tip task_201305131710_0007_m_000430, for tracker 'tracker_ZC329-0001.labs.edu.au:127.0.0.1/127.0.0.1:1113'
4

1 に答える 1

0

各クラスター ノードの構成を確認することができます。

tracker_HN301-1657.labs.edu.au:127.0.0.1/127.0.0.1:3896

jobTracker は、ループ バック アドレスを介して TaskTracker ノードに接続しようとしています。各ノードの /etc/hosts ファイルの内容をチェックして、それらが正しいことを確認します (また、DNS ルックアップのコストを回避できるように、クラスター内の他の各ノードについて知っていることが望ましいです)。

これがあなたの問題の原因だと言っているわけではありませんが、それは確かに正しくありません。

于 2013-05-20T10:30:13.993 に答える