基于Python语言NLTK的评论情感分析

Question

我有一个csv数据文件，包含列'笔记'满意的答案在希伯来语。你知道吗

我想使用情绪分析，以便为数据中的每个单词或bigrm分配一个分数，并使用logistic回归获得正/负概率。你知道吗

到目前为止我的代码是：

PYTHONIOENCODING="UTF-8"  
df= pd.read_csv('keep.csv', encoding='utf-8' , usecols=['notes'])

txt = df.notes.str.lower().str.replace(r'\|', ' ').str.cat(sep=' ')
words = nltk.tokenize.word_tokenize(txt)
tokens=[word.lower() for word in words if word.isalpha()]
bigrm = list(nltk.bigrams(tokens))

word_index = {}
current_index = 0
    for token in tokens:
    if token not in word_index:
        word_index[token] = current_index
        current_index += 1

def tokens_to_vector(tokens, label):
    x = np.zeros(len(word_index) + 1) 
    for t in tokens:
        i = word_index[t]
        x[i] += 1
    x = x / x.sum() 
    x[-1] = label
    return x

N= len(word_index)
data = np.zeros((N, len(word_index) + 1))
i = 0
for token in tokens:
xy = tokens_to_vector(tokens, 1)
data[i,:] = xy
i += 1

这个循环不起作用。 如何生成数据，然后接收每个bigrm的正/负概率？你知道吗

Answer 1

你的代码片段正确吗？所有for循环都需要缩进。你知道吗

df= pd.read_csv('keep.csv', encoding='utf-8' , usecols=['notes'])

txt = df.notes.str.lower().str.replace(r'\|', ' ').str.cat(sep=' ')
words = nltk.tokenize.word_tokenize(txt)
tokens=[word.lower() for word in words if word.isalpha()]
bigrm = list(nltk.bigrams(tokens))

word_index = {}
current_index = 0
    for token in tokens:
        if token not in word_index:
            word_index[token] = current_index
            current_index += 1

def tokens_to_vector(tokens, label):
    x = np.zeros(len(word_index) + 1) 
    for t in tokens:
        i = word_index[t]
        x[i] += 1
    x = x / x.sum() 
    x[-1] = label
    return x

N= len(word_index)
data = np.zeros((N, len(word_index) + 1))
i = 0
for token in tokens:
    xy = tokens_to_vector(tokens, 1)
    data[i,:] = xy
    i += 1```