如何使用pandas/sklearn删除停止短语/停止ngram（多单词字符串）？

Question

我想防止某些短语进入我的模型。例如，我想阻止“红玫瑰”进入我的分析。我知道如何添加Adding words to scikit-learn's CountVectorizer's stop list中给出的单个停止词，方法是：

from sklearn.feature_extraction import text
additional_stop_words=['red','roses']

然而，这也会导致诸如“红郁金香”或“蓝玫瑰”之类的其他ngram检测不到。在

我正在构建一个TfidfVectorizer作为我的模型的一部分，我意识到我需要的处理可能必须在这一阶段之后进入，但我不确定如何做到这一点。在

我的最终目标是在一段文本上做主题建模。下面是我正在处理的代码段（几乎直接从https://de.dariah.eu/tatom/topic_model_python.html#index-0借用的）：

^{pr2}$

编辑

示例数据帧（我尝试插入尽可能多的边缘情况），df：

   Content
0  I like red roses as much as I like blue tulips.
1  It would be quite unusual to see red tulips, but not RED ROSES
2  It is almost impossible to find blue roses
3  I like most red flowers, but roses are my favorite.
4  Could you buy me some red roses?
5  John loves the color red. Roses are Mary's favorite flowers.

Answer 1

在将df传递给mod_vectorizer之前，您应该使用类似以下命令：

df=["I like red roses as much as I like blue tulips.",
"It would be quite unusual to see red tulips, but not RED ROSES",
"It is almost impossible to find blue roses",
"I like most red flowers, but roses are my favorite.",
"Could you buy me some red roses?",
"John loves the color red. Roses are Mary's favorite flowers."]

df=[ i.lower() for i in df]
df=[i if 'red roses' not in i else i.replace('red roses','') for i in df]

如果您要检查的不止“红玫瑰”，请将上面最后一行替换为：

^{pr2}$