Hadoop フレームワークと map reduce 抽象化は初めてです。
基本的には、巨大なテキストファイル(「,」で区切られた)の中で最小の数字を見つけることを考えました
だから、ここに私のコードmapper.pyがあります
#!/usr/bin/env python
import sys
# input comes from STDIN (standard input)
for line in sys.stdin:
# remove leading and trailing whitespace
line = line.strip()
# split the line into words
numbers = line.split(",")
# increase counters
for number in numbers:
# write the results to STDOUT (standard output);
# what we output here will be the input for the
# Reduce step, i.e. the input for reducer.py
#
# tab-delimited; the trivial word count is 1
print '%s\t%s' % (number, 1)
減速機
#!/usr/bin/env python
from operator import itemgetter
import sys
smallest_number = sys.float_info.max
for line in sys.stdin:
# remove leading and trailing whitespace
line = line.strip()
# parse the input we got from mapper.py
number, count = line.split('\t', 1)
try:
number = float(number)
except ValueError:
continue
if number < smallest_number:
smallest_number = number
print smallest_number <---- i think the error is here... there is no key value thingy
print smallest_number
私が得るエラー:
12/10/04 12:07:22 ERROR streaming.StreamJob: Job not successful. Error: NA
12/10/04 12:07:22 INFO streaming.StreamJob: killJob...
Streaming Command Failed!