然后,检查同义词是否共享部分词干。本质上是if "gas" is in "gasolin",反之亦然。这就足够了,因为你只比较你的同义词
import spacy
import itertools
from nltk.stem.porter import *
threshold = 0.6
#compare the stems of the synonyms
stemmer = PorterStemmer()
def compare_stems(a, b):
if stemmer.stem(a) in stemmer.stem(b):
return True
if stemmer.stem(b) in stemmer.stem(a):
return True
return False
candidate_synonyms = {}
#add a candidate to the candidate dictionary of sets
def add_to_synonym_dict(a,b):
if a not in candidate_synonyms:
if b not in candidate_synonyms:
candidate_synonyms[a] = {a, b}
return
a, b = b,a
candidate_synonyms[a].add(b)
nlp = spacy.load('en_core_web_lg')
text = u'The price of gasoline has risen. "Gas" is a colloquial form of the word gasoline in North American English. Conversely in BE the term would be petrol. A gaseous state has nothing to do with oil.'
words = nlp(text)
#compare every word with every other word, if they are similar
for a, b in itertools.combinations(words, 2):
#check if one of the word pairs are stopwords or punctuation
if a.is_stop or b.is_stop or a.is_punct or b.is_punct:
continue
if a.similarity(b) > threshold:
if compare_stems(a.text.lower(), b.text.lower()):
add_to_synonym_dict(a.text.lower(), b.text.lower())
print(candidate_synonyms)
#output: {'gasoline': {'gas', 'gasoline'}}
我建议采取两步办法:
首先,通过比较单词嵌入(仅非停止词)查找同义词。这应该删除类似的书面单词,它们意味着其他东西,例如
gasoline
和gaseous
然后,检查同义词是否共享部分词干。本质上是
if "gas" is in "gasolin"
,反之亦然。这就足够了,因为你只比较你的同义词然后,您可以根据同义词在文本中的外观来计算候选同义词
注意:我偶然选择了0.6同义词的阈值。您可能会测试哪个阈值适合您的任务。另外,我的代码只是一个快速而肮脏的例子,这可以做得更干净。 `
相关问题 更多 >
编程相关推荐