<p>为什么余弦相似性和TFIDF矢量器对您不起作用</p>
<p>我试过了,它与以下代码一起工作:</p>
<pre><code>import pandas as pd
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
df = pd.DataFrame(columns=["ID","DESCRIPTION"], data=np.matrix([[10,"Cancel ASN WMS Cancel ASN"],
[11,"MAXPREDO Validation is corect"],
[12,"Move to QC"],
[13,"Cancel ASN WMS Cancel ASN"],
[14,"MAXPREDO Validation is right"],
[15,"Verify files are sent every hours for this interface from Optima"],
[16,"MAXPREDO Validation are correct"],
[17,"Move to QC"],
[18,"Verify files are not sent"]
]))
corpus = list(df["DESCRIPTION"].values)
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(corpus)
threshold = 0.4
for x in range(0,X.shape[0]):
for y in range(x,X.shape[0]):
if(x!=y):
if(cosine_similarity(X[x],X[y])>threshold):
print(df["ID"][x],":",corpus[x])
print(df["ID"][y],":",corpus[y])
print("Cosine similarity:",cosine_similarity(X[x],X[y]))
print()
</code></pre>
<p>阈值也可以调整,但如果阈值为0.9,则不会产生所需的结果</p>
<p>阈值为0.4的输出为:</p>
<pre><code>10 : Cancel ASN WMS Cancel ASN
13 : Cancel ASN WMS Cancel ASN
Cosine similarity: [[1.]]
11 : MAXPREDO Validation is corect
14 : MAXPREDO Validation is right
Cosine similarity: [[0.64183024]]
12 : Move to QC
17 : Move to QC
Cosine similarity: [[1.]]
15 : Verify files are sent every hours for this interface from Optima
18 : Verify files are not sent
Cosine similarity: [[0.44897995]]
</code></pre>
<p>阈值为0.39时,所有预期句子都是输出中的特征,但也可以找到另外一对具有索引[15,18]的句子:</p>
<pre><code>10 : Cancel ASN WMS Cancel ASN
13 : Cancel ASN WMS Cancel ASN
Cosine similarity: [[1.]]
11 : MAXPREDO Validation is corect
14 : MAXPREDO Validation is right
Cosine similarity: [[0.64183024]]
11 : MAXPREDO Validation is corect
16 : MAXPREDO Validation are correct
Cosine similarity: [[0.39895808]]
12 : Move to QC
17 : Move to QC
Cosine similarity: [[1.]]
14 : MAXPREDO Validation is right
16 : MAXPREDO Validation are correct
Cosine similarity: [[0.39895808]]
15 : Verify files are sent every hours for this interface from Optima
18 : Verify files are not sent
Cosine similarity: [[0.44897995]]
</code></pre>