3

I seriously hate to post a question about an entire chunk of code, but I've been working on this for the past 3 hours and I can't wrap my head around what is happening. I have approximately 600 tweets I am retrieving from a CSV file with varying score values (between -2 to 2) reflecting the sentiment towards a presidential candidate.

However, when I run this training sample on any other data, only one value is returned (positive). I have checked to see if the scores were being added correctly and they are. It just doesn't make sense to me that 85,000 tweets would all be rated "positive" from a diverse training set of 600. Does anyone know what is happening here? Thanks!

import nltk
import csv

tweets = []
import ast
with open('romney.csv', 'rb') as csvfile:
    mycsv = csv.reader(csvfile)
    for row in mycsv:
        tweet = row[1]
        try:
            score = ast.literal_eval(row[12])
            if score > 0:
                print score
                print tweet
                tweets.append((tweet,"positive"))

        elif score < 0:
            print score
            print tweet
            tweets.append((tweet,"negative"))
    except ValueError:
        tweet = ""

def get_words_in_tweets(tweets):
    all_words = []
    for (words, sentiment) in tweets:
      all_words.extend(words)
    return all_words

def get_word_features(wordlist):
    wordlist = nltk.FreqDist(wordlist)
    word_features = wordlist.keys()
    return word_features

def extract_features(document):
    document_words = set(document)
    features = {}
    for word in word_features:
    features['contains(%s)' % word] = (word in document_words)
    return features

word_features = get_word_features(get_words_in_tweets(tweets))
training_set = nltk.classify.apply_features(extract_features, tweets)
classifier = nltk.NaiveBayesClassifier.train(training_set)
c = 0
with open('usa.csv', "rU") as csvfile:
    mycsv = csv.reader(csvfile)
    for row in mycsv:
        try:
            tweet = row[0]
            c = c + 1
                    print classifier.classify(extract_features(tweet.split()))                                                                                                                                                                                     
        except IndexError:
            tweet = ""
4

1 に答える 1

2

Naive Bayes Classifier は、通常、文書内に出現する単語を評価し、単語がないことを無視する場合に最適に機能します。使っているので

features['contains(%s)' % word] = (word in document_words)

各ドキュメントは、ほとんどの場合、値 = False の機能によって表されます。

代わりに次のようなものを試してください:

if word in document_words:
   features['contains(%s)' % word] = True

(辞書内のすべての単語をループするよりも効率的なものにforループを変更し、代わりにドキュメント内の単語をループする必要があります)。

于 2013-02-27T20:49:56.507 に答える