CountVectorizer不尊重正则表达式

Question

我使用以下代码获取文档术语矩阵：

from nltk.stem import SnowballStemmer
from sklearn.feature_extraction.text import CountVectorizer

stemmer = SnowballStemmer("english", ignore_stopwords=True)


class StemmedCountVectorizer(CountVectorizer):
    def build_analyzer(self):
        analyzer = super(StemmedCountVectorizer, self).build_analyzer()
        return lambda doc: ([stemmer.stem(w) for w in analyzer(doc)])

stemmed_count_vect = StemmedCountVectorizer(stop_words='english', 
                                            ngram_range=(1,1), 
                                            token_pattern=r'\b\w+\b', 
                                            min_df=1, 
                                            max_df=0.6)

不过，我还是得到了这样的东西：

20009
2000mile
20011
20017
2001â
2007s
20081007
200h

我该怎么解决这个问题

Answer 1

此模式token_pattern=r'\b\w+\b'表示它希望单词边界之间有一个或多个\w字符类的成员。这个角色类

[m]atches Unicode word characters; this includes most characters that can be part of a word in any language, as well as numbers and the underscore.

所以在我看来，你需要更少的字符类（省略数字作为开始）