Spacy短语匹配器不匹配一个关键字，而是匹配字符串中的所有关键字

Question

我试图解决根据关键字将文本分类到桶中的任务。当我需要将文本与一个或多个关键字（因此其中一个关键字应在文本中）进行匹配时，这是相当容易的，但是当我需要确保字符串中存在多个关键字时，我很难理解如何进行匹配

下面是一个小样本。假设我的dfArticles是一个pandas数据框架，它有一列Text与我试图匹配的文本文章：

dfArticles['Text']
Out[2]: 
0       (Reuters) - Major Middle Eastern markets ended...
1       MIDEAST STOCKS-Oil price fall hurts major Gulf...
2       DUBAI, 21st September, 2020 (WAM) -- The Minis...
3       DUBAI, (UrduPoint / Pakistan Point News / WAM ...
4       Brent crude was down 99 cents or 2.1% at $42.2.

我们还可以说，我的dataframedfTopics包含我试图匹配的关键字列表以及与关键字关联的bucket：

dfTopics
Out[3]: 
            Topic              Keywords
0     Regulations                   law
1     Regulations            regulatory
2     Regulations            regulation
3     Regulations           legislation
4     Regulations                 rules
5          Talent            capability
6          Talent             workforce

当我只需要检查文本是否匹配其中一个关键字时，很简单：

def prep_match_patterns(dfTopics):
    
    matcher = PhraseMatcher(nlp.vocab, attr="LOWER")
    
    for topic in dfTopics['Topic'].unique():
        keywords = dfTopics.loc[dfTopics['Topic'] == topic, 'Keywords'].to_list()
        patterns_topic = [nlp.make_doc(text) for text in keywords]
        matcher.add(topic, None, *patterns_topic)
    return matcher

然后，我可以很容易地用一个镜头检查文本属于哪个桶：

nlp = spacy.load("en_core_web_lg")
nlp.disable_pipes(["parser"])
# extract the sentences from the documents
nlp.add_pipe(nlp.create_pipe('sentencizer'))

matcher = prep_match_patterns(dfTopics)


dfResults = pd.DataFrame([],columns=['ArticleID', 'Topic'])


articles = []
topics = []


for index, row in tqdm(dfArticles.iterrows(), total=len(dfArticles)):
    doc = nlp(row['Text'])
    matches = matcher(doc)
    if len(matches)<1:
        continue
    else:
        for match_id, start, end in matches:
            string_id = nlp.vocab.strings[match_id]  # Get string representation
            articles.append(row['ID'])
            topics.append(string_id)
    
dfResults['ArticleID'] = articles
dfResults['Topic'] = topics


dfResults.drop_duplicates(inplace=True)

但现在的诀窍是-有时要将文本分类到bucket中，我需要确保它同时匹配多个关键字

假设我有一个名为“医疗系统上下文”的新主题，要将文本放入这个桶中，我需要文本中包含所有3个子字符串：“碎片化”和“批准流程”以及“药物”。顺序并不重要，但所有三个关键字都需要存在。用短语匹配器有什么办法吗

Answer 1

我觉得你太复杂了。您可以使用简单的python实现您想要的

假设我们有：

df_topics
    Topic   Keywords
0   Regulations law
1   Regulations regulatory
2   Regulations regulation
3   Regulations legislation
4   Regulations rules
5   Talent  capability
6   Talent  workforce

然后，您可以将主题关键字组织到词典中：

topics = df_topics.groupby("Topic")["Keywords"].agg(lambda x: x.to_list()).to_dict()
topics
{'Regulations': ['law', 'regulatory', 'regulation', 'legislation', 'rules'],
 'Talent': ['capability', 'workforce']}

最后，定义一个func以匹配关键字：

def textToTopic(text, topics):
    t = []
    for k,v in topics.items():
        if all([topic in text.split() for topic in v]):
            t.append(k)
    return t

演示：

textToTopic("law regulatory regulation rules legislation workforce", topics)
['Regulations']

---

textToTopic("law regulatory regulation rules legislation workforce capability", topics)
['Regulations', 'Talent']

您可以将此函数应用于df中的文本