如何将句子加载到Python gensim中？

Question

我试图使用Python中自然语言处理库中的gensim模块。

文件上说要初始化模型：

from gensim.models import word2vec
model = Word2Vec(sentences, size=100, window=5, min_count=5, workers=4)

输入语句需要什么格式？我有原始文本

"the quick brown fox jumps over the lazy dogs"
"Then a cop quizzed Mick Jagger's ex-wives briefly."
etc.

我还需要在word2fec中进行哪些额外的处理？

---

更新：以下是我所做的尝试。当它加载句子时，我什么也得不到。

>>> sentences = ['the quick brown fox jumps over the lazy dogs',
             "Then a cop quizzed Mick Jagger's ex-wives briefly."]
>>> x = word2vec.Word2Vec()
>>> x.build_vocab([s.encode('utf-8').split( ) for s in sentences])
>>> x.vocab
{}

Answer 1

就像alKid指出的那样，使它成为utf-8。

谈论另外两件你可能需要担心的事情。

1. 输入太大，正在从文件加载。
2. 删除句子中的停止词。

您可以执行以下操作，而不是将大列表加载到内存中：

import nltk, gensim
class FileToSent(object):    
    def __init__(self, filename):
        self.filename = filename
        self.stop = set(nltk.corpus.stopwords.words('english'))

    def __iter__(self):
        for line in open(self.filename, 'r'):
        ll = [i for i in unicode(line, 'utf-8').lower().split() if i not in self.stop]
        yield ll

然后

sentences = FileToSent('sentence_file.txt')
model = gensim.models.Word2Vec(sentences=sentences, window=5, min_count=5, workers=4, hs=1)