<p>这是一个很好的例子,可以使用比string matching=更聪明的方法)</p>
<p>让我们考虑一下:</p>
<ul>
<li><p>有没有办法把每个单词都转换成向量形式(即浮点数组)?</p></li>
<li><p>有没有一种方法可以将每个句子转换成相同的向量形式(例如,一个与单词向量形式大小相同的浮点数组)?</p></li>
</ul>
<hr/>
<p>首先,让我们从你的句子列表中找出所有可能的单词(我们称之为语料库):</p>
<pre><code>>>> from itertools import chain
>>> s1 = "Lecture was engaging"
>>> s2 = "Tutor is very nice and active"
>>> s3 = "The content of lecture was too much for 2 hours."
>>> s4 = "Exam seem to be too difficult compare with weekly lab."
>>> list(map(word_tokenize, [s1, s2, s3, s4]))
[['Lecture', 'was', 'engaging'], ['Tutor', 'is', 'very', 'nice', 'and', 'active'], ['The', 'content', 'of', 'lecture', 'was', 'too', 'much', 'for', '2', 'hours', '.'], ['Exam', 'seem', 'to', 'be', 'too', 'difficult', 'compare', 'with', 'weekly', 'lab', '.']]
>>> vocab = sorted(set(token.lower() for token in chain(*list(map(word_tokenize, [s1, s2, s3, s4])))))
>>> vocab
['.', '2', 'active', 'and', 'be', 'compare', 'content', 'difficult', 'engaging', 'exam', 'for', 'hours', 'is', 'lab', 'lecture', 'much', 'nice', 'of', 'seem', 'the', 'to', 'too', 'tutor', 'very', 'was', 'weekly', 'with']
</code></pre>
<p>现在让我们用词汇表中单词的索引将4个关键词表示为向量:</p>
^{pr2}$
<p>类似地,我们循环每个句子并将其转换为向量形式:</p>
<pre><code>>>> [token.lower() for token in word_tokenize(s1)]
['lecture', 'was', 'engaging']
>>> s1_tokens = [token.lower() for token in word_tokenize(s1)]
>>> s1_vec = [1 if token in s1_tokens else 0 for token in vocab]
>>> s1_vec
[0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0]
</code></pre>
<p>对所有句子重复相同的内容:</p>
<pre><code>>>> s2_tokens = [token.lower() for token in word_tokenize(s2)]
>>> s3_tokens = [token.lower() for token in word_tokenize(s3)]
>>> s4_tokens = [token.lower() for token in word_tokenize(s4)]
>>> s2_vec = [1 if token in s2_tokens else 0 for token in vocab]
>>> s3_vec = [1 if token in s3_tokens else 0 for token in vocab]
>>> s4_vec = [1 if token in s4_tokens else 0 for token in vocab]
>>> s2_vec
[0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0]
>>> s3_vec
[1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 1, 1, 0, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0]
>>> s4_vec
[1, 0, 0, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 1, 1]
</code></pre>
<p>现在,给定句子和单词的向量形式,您可以使用相似函数,例如<a href="https://en.wikipedia.org/wiki/Cosine_similarity" rel="nofollow noreferrer">cosine similarity</a>:</p>
<pre><code>>>> from numpy import dot
>>> from numpy.linalg import norm
>>>
>>> cos_sim = lambda x, y: dot(x,y)/(norm(x)*norm(y))
>>> cos_sim(s1_vec, lecture)
0.5773502691896258
>>> cos_sim(s1_vec, lab)
0.0
>>> cos_sim(s1_vec, exam)
0.0
>>> cos_sim(s1_vec, tutor)
0.0
</code></pre>
<p>现在,更系统地进行:</p>
<pre><code>>>> topics = {'lecture': lecture, 'lab': lab, 'exam': exam, 'tutor':tutor}
>>> topics
{'lecture': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
'lab': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
'exam': [0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
'tutor': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0]}
>>> sentences = {'s1':s1_vec, 's2':s2_vec, 's3':s3_vec, 's4':s4_vec}
>>> for s_num, s_vec in sentences.items():
... print(s_num)
... for name, topic_vec in topics.items():
... print('\t', name, cos_sim(s_vec, topic_vec))
...
s1
lecture 0.5773502691896258
lab 0.0
exam 0.0
tutor 0.0
s2
lecture 0.0
lab 0.0
exam 0.0
tutor 0.4082482904638631
s3
lecture 0.30151134457776363
lab 0.0
exam 0.0
tutor 0.0
s4
lecture 0.0
lab 0.30151134457776363
exam 0.30151134457776363
tutor 0.0
</code></pre>
<hr/>
<p>我想你明白了。但我们看到,s4实验室和s4考试的分数仍然持平。所以问题就变成了,“有没有办法让他们分道扬镳?”你会跳进兔子洞里:</p>
<ul>
<li><p>如何最好地将句子/单词表示为固定大小的向量?</p></li>
<li><p>比较“主题”和“主题”的相似性?</p></li>
<li><p>什么是“话题”?向量实际上代表什么?</p></li>
</ul>
<p>上面的答案是通常所说的一个热向量来表示单词/句子。比简单地比较字符串来“识别与主题相关的句子”要复杂得多(也称为文档聚类/分类)。E、 一个文件/句子可以有多个主题吗?在</p>
<p>查找这些关键词,进一步了解“自然语言处理”、“文档分类”、“机器学习”等问题。同时,如果你不介意的话,我想这个问题已经很接近了,因为<strong>“太宽泛”</strong>。在</p>