私は Cloudera と Hadoop の両方に不慣れで、Cloudera WordCount 1.0 の例 (part-00000) の出力は空です。私が使用している手順とファイルはこちらです。バージョンについても同様に、役立つジョブログ情報を提供したいと思います-それらを見つける場所についてのガイダンスが必要なだけです。以下は、ジョブの出力とソースです。書き込まれた他の部分 (part-00001 から part-00011) のうち、空でない部分は、part-00001 (Bye 1)、part-00002 (Hadoop 2)、part-00004 (Goodbye 1)、part-00005 (World 2)、part-00009 (Hello 2)。どんな助けでも素晴らしいでしょう。
コマンドと出力は次のとおりです。
[me@server ~]$ hadoop fs -cat /user/me/wordcount/input/file0
Hello World Bye World
[me@server ~]$ hadoop fs -cat /user/me/wordcount/input/file1
Hello Hadoop Goodbye Hadoop
[me@server ~]$ hadoop jar wordcount.jar org.myorg.WordCount /user/me/wordcount/input /user/me/wordcount/output
13/11/12 10:39:41 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/11/12 10:39:41 INFO mapred.FileInputFormat: Total input paths to process : 2
13/11/12 10:39:42 INFO mapred.JobClient: Running job: job_201311051201_0014
13/11/12 10:39:43 INFO mapred.JobClient: map 0% reduce 0%
13/11/12 10:39:49 INFO mapred.JobClient: map 33% reduce 0%
13/11/12 10:39:52 INFO mapred.JobClient: map 67% reduce 0%
13/11/12 10:39:53 INFO mapred.JobClient: map 100% reduce 0%
13/11/12 10:39:58 INFO mapred.JobClient: map 100% reduce 25%
13/11/12 10:40:01 INFO mapred.JobClient: map 100% reduce 100%
13/11/12 10:40:04 INFO mapred.JobClient: Job complete: job_201311051201_0014
13/11/12 10:40:04 INFO mapred.JobClient: Counters: 33
13/11/12 10:40:04 INFO mapred.JobClient: File System Counters
13/11/12 10:40:04 INFO mapred.JobClient: FILE: Number of bytes read=313
13/11/12 10:40:04 INFO mapred.JobClient: FILE: Number of bytes written=2695420
13/11/12 10:40:04 INFO mapred.JobClient: FILE: Number of read operations=0
13/11/12 10:40:04 INFO mapred.JobClient: FILE: Number of large read operations=0
13/11/12 10:40:04 INFO mapred.JobClient: FILE: Number of write operations=0
13/11/12 10:40:04 INFO mapred.JobClient: HDFS: Number of bytes read=410
13/11/12 10:40:04 INFO mapred.JobClient: HDFS: Number of bytes written=41
13/11/12 10:40:04 INFO mapred.JobClient: HDFS: Number of read operations=18
13/11/12 10:40:04 INFO mapred.JobClient: HDFS: Number of large read operations=0
13/11/12 10:40:04 INFO mapred.JobClient: HDFS: Number of write operations=24
13/11/12 10:40:04 INFO mapred.JobClient: Job Counters
13/11/12 10:40:04 INFO mapred.JobClient: Launched map tasks=3
13/11/12 10:40:04 INFO mapred.JobClient: Launched reduce tasks=12
13/11/12 10:40:04 INFO mapred.JobClient: Data-local map tasks=3
13/11/12 10:40:04 INFO mapred.JobClient: Total time spent by all maps in occupied slots (ms)=16392
13/11/12 10:40:04 INFO mapred.JobClient: Total time spent by all reduces in occupied slots (ms)=61486
13/11/12 10:40:04 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
13/11/12 10:40:04 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
13/11/12 10:40:04 INFO mapred.JobClient: Map-Reduce Framework
13/11/12 10:40:04 INFO mapred.JobClient: Map input records=2
13/11/12 10:40:04 INFO mapred.JobClient: Map output records=8
13/11/12 10:40:04 INFO mapred.JobClient: Map output bytes=82
13/11/12 10:40:04 INFO mapred.JobClient: Input split bytes=357
13/11/12 10:40:04 INFO mapred.JobClient: Combine input records=8
13/11/12 10:40:04 INFO mapred.JobClient: Combine output records=6
13/11/12 10:40:04 INFO mapred.JobClient: Reduce input groups=5
13/11/12 10:40:04 INFO mapred.JobClient: Reduce shuffle bytes=649
13/11/12 10:40:04 INFO mapred.JobClient: Reduce input records=6
13/11/12 10:40:04 INFO mapred.JobClient: Reduce output records=5
13/11/12 10:40:04 INFO mapred.JobClient: Spilled Records=12
13/11/12 10:40:04 INFO mapred.JobClient: CPU time spent (ms)=15650
13/11/12 10:40:04 INFO mapred.JobClient: Physical memory (bytes) snapshot=3594293248
13/11/12 10:40:04 INFO mapred.JobClient: Virtual memory (bytes) snapshot=18375352320
13/11/12 10:40:04 INFO mapred.JobClient: Total committed heap usage (bytes)=6497697792
13/11/12 10:40:04 INFO mapred.JobClient: org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
13/11/12 10:40:04 INFO mapred.JobClient: BYTES_READ=50
[me@server ~]$ hadoop fs -cat /user/me/wordcount/output/part-00000
[me@server ~]$ hdfs dfs -ls -R /user/me/wordcount/output
-rw-r--r-- 3 me me 0 2013-11-12 10:40 /user/me/wordcount/output/_SUCCESS
drwxr-xr-x - me me 0 2013-11-12 10:39 /user/me/wordcount/output/_logs
drwxr-xr-x - me me 0 2013-11-12 10:39 /user/me/wordcount/output/_logs/history
-rw-r--r-- 3 me me 67134 2013-11-12 10:40 /user/me/wordcount/output/_logs/history/job_201311051201_0014_1384270782432_me_wordcount
-rw-r--r-- 3 me me 81866 2013-11-12 10:39 /user/me/wordcount/output/_logs/history/job_201311051201_0014_conf.xml
-rw-r--r-- 3 me me 0 2013-11-12 10:39 /user/me/wordcount/output/part-00000
-rw-r--r-- 3 me me 6 2013-11-12 10:39 /user/me/wordcount/output/part-00001
-rw-r--r-- 3 me me 9 2013-11-12 10:39 /user/me/wordcount/output/part-00002
-rw-r--r-- 3 me me 0 2013-11-12 10:39 /user/me/wordcount/output/part-00003
-rw-r--r-- 3 me me 10 2013-11-12 10:39 /user/me/wordcount/output/part-00004
-rw-r--r-- 3 me me 8 2013-11-12 10:39 /user/me/wordcount/output/part-00005
-rw-r--r-- 3 me me 0 2013-11-12 10:39 /user/me/wordcount/output/part-00006
-rw-r--r-- 3 me me 0 2013-11-12 10:39 /user/me/wordcount/output/part-00007
-rw-r--r-- 3 me me 0 2013-11-12 10:39 /user/me/wordcount/output/part-00008
-rw-r--r-- 3 me me 8 2013-11-12 10:39 /user/me/wordcount/output/part-00009
-rw-r--r-- 3 me me 0 2013-11-12 10:39 /user/me/wordcount/output/part-00010
-rw-r--r-- 3 me me 0 2013-11-12 10:39 /user/me/wordcount/output/part-00011
[me@server ~]$
ソースは次のとおりです。
package org.myorg;
import java.io.IOException;
import java.util.*;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;
public class WordCount {
public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
output.collect(word, one);
}
}
}
public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
int sum = 0;
while (values.hasNext()) {
sum += values.next().get();
}
output.collect(key, new IntWritable(sum));
}
}
public static void main(String[] args) throws Exception {
JobConf conf = new JobConf(WordCount.class);
conf.setJobName("wordcount");
conf.setOutputKeyClass(Text.class);
conf.setOutputValueClass(IntWritable.class);
conf.setMapperClass(Map.class);
conf.setCombinerClass(Reduce.class);
conf.setReducerClass(Reduce.class);
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
FileInputFormat.setInputPaths(conf, new Path(args[0]));
FileOutputFormat.setOutputPath(conf, new Path(args[1]));
JobClient.runJob(conf);
}
}