python中文本到向量的转换

Question

import tokenize
tags = [
  "python, tools",
  "linux, tools, ubuntu",
  "distributed systems, linux, networking, tools",
]
from sklearn.feature_extraction.text import CountVectorizer
vec = CountVectorizer(tokenizer=tokenize)
data = vec.fit_transform(tags).toarray()
print data

我正在尝试将文本转换为矢量。但我面临以下错误

^{pr2}$

我也尝试过导入其他库。但似乎什么都不管用。我怎样才能改正呢？在

Answer 1

不完全是一个解决方案更多的是从主页nltk.org网站公司名称：

>>> import nltk
 >>> sentence = """At eight o'clock on Thursday morning
 ... Arthur didn't feel very good."""
>>> tokens = nltk.word_tokenize(sentence)
>>> tokens
['At', 'eight', "o'clock", 'on', 'Thursday', 'morning',
'Arthur', 'did', "n't", 'feel', 'very', 'good', '.']

希望这有帮助