2

組み込みの Pig を使用して、グラフ アルゴリズムを実装しています。ローカルモードでは正常に動作します。しかし、完全に分散された Hadoop クラスターでは、次のようなエラー メッセージが常に表示されます (最後の数行を参照してください)。

2012-11-23 22:00:00,651 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job4116346741117365374.jar
2012-11-23 22:00:09,418 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job4116346741117365374.jar created
2012-11-23 22:00:09,423 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up multi store job
2012-11-23 22:00:09,431 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=296
2012-11-23 22:00:09,431 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Neither PARALLEL nor default parallelism is set for this job. Setting number of reducers to 1
2012-11-23 22:00:09,442 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2012-11-23 22:00:09,949 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job null has failed! Stop running all dependent jobs
2012-11-23 22:00:09,949 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2012-11-23 22:00:09,992 [main] ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR 6015: During execution, encountered a Hadoop error.
2012-11-23 22:00:09,993 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2012-11-23 22:00:09,994 [main] INFO org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:

HadoopVersion    PigVersion    UserId    StartedAt    FinishedAt Features
0.20.1    0.10.0    jierus    2012-11-23 21:52:38    2012-11-23 22:00:09    HASH_JOIN,GROUP_BY,DISTINCT,FILTER,UNION

Some jobs have failed! Stop running all dependent jobs
Failed Jobs:
JobId    Alias    Feature    Message    Outputs
N/A    vec_comp,vec_comp_final,vec_comp_tmp HASH_JOIN,MULTI_QUERY    Message: java.io.FileNotFoundException: File /tmp/Job4116346741117365374.jar does not exist.
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:361)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:192)
    at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1184)
    at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1160)
    at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1132)

私のコードまたは操作のどの部分が間違っているか誰か知っていますか?

4

1 に答える 1

0

pig のジョブ トラッカーを指定していないようです (hdfs を指定するだけでは不十分です!)。例えば

<property>
    <name>mapred.job.tracker</name>
    <value>10.xx.xx.99:9001</value>
</property>
于 2013-07-17T06:01:38.220 に答える