0

実際、私はhadoopとpythonも初めてです....だから私の疑問は、hadoopでpythonスクリプトを実行する方法です.....また、pythonを使用してwordcountプログラムを書いていました..だから、このスクリプトを実行できますかmap reduce を使用して .... 実際にコードを書きました。以下のような出力を見ることができます。 Darkness 1 Heaven 2 It 3 Light 4 age 5 age 6 all 7 all 8 authority 9 before 10 before 11 being 12 信念 13 best 14 comparison 15程度 16 絶望 17 直接 18 直接 19

It is counting number of words in a list..but whati have to achieve is grouping and deleting the duplicates and also count number of times of its occurrences  ..... 

Below is my code . can somebody please tell me where i have done the mistake

********************************************************
   Wordcount.py
********************************************************

import urllib2
import random
from operator import itemgetter

current_word = {}
current_count = 0
story = 'http://sixty-north.com/c/t.txt'
request = urllib2.Request(story)
response = urllib2.urlopen(request)
each_word = []
words = None
count = 1
same_words ={}
word = []
""" looping the entire file """
for line in response:
    line_words = line.split()
    for word in line_words:  # looping each line and extracting words
        each_word.append(word)
        random.shuffle(each_word)
        Sort_word = sorted(each_word)
for words in Sort_word:
    same_words = words.lower(),int(count)
    #print same_words
    #print words
    if not words in current_word :
        current_count = current_count +1
        print '%s\t%s' % (words, current_count)
    else:
        current_count = 1
        #if Sort_word == words.lower():
            #current_count += count
current_count = count
current_word = word
        #print '2. %s\t%s' % (words, current_count)
4

1 に答える 1

0

Python ベースの MR タスクを実行するには、以下を参照してください。

http://hadoop.apache.org/docs/r1.1.2/streaming.html http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/

Hadoop が Python スクリプトを実行できるようにするには、Mapper - Reducer の観点からコードを設計する必要があります。コードを書き始める前に、Map-Reduce プログラミング パラダイムを読んでください。MR プログラミング パラダイムと、問題を解決する際の {Key , value } ペアの役割を理解することが重要です。

#Modified your above code to generate the required output
import urllib2
import random
from operator import itemgetter

current_word = {}
current_count = 0
story = 'http://sixty-north.com/c/t.txt'
request = urllib2.Request(story)
response = urllib2.urlopen(request)
each_word = []
words = None
count = 1
same_words ={}
word = []
""" looping the entire file """
#Collect All the words into a list
for line in response:
    #print "Line = " , line
    line_words = line.split()
    for word in line_words:  # looping each line and extracting words
        each_word.append(word)

#for every word collected, in dict same_words
#if a key exists, such that key == word then increment Mapping Value by 1
# Else add word as new key with mapped value as 1
for words in each_word:
    if words.lower() not in same_words.keys() :
        same_words[words.lower()]=1
    else:
        same_words[words.lower()]=same_words[words.lower()]+1

for each in same_words.keys():
    print "word = ",each, ", count = ",same_words[each]
于 2013-10-30T05:20:04.673 に答える