回答此问题可获得 20 贡献值,回答如果被采纳可获得 50 分。
<p>我试图调整这段代码(source found <a href="http://stevenloria.com/finding-important-words-in-a-document-using-tf-idf/" rel="nofollow noreferrer">here</a>)来遍历一个文件目录,而不是对输入进行硬编码。在</p>
<pre><code>#!/usr/bin/python
# -*- coding: utf-8 -*-
from __future__ import division, unicode_literals
import math
from textblob import TextBlob as tb
def tf(word, blob):
return blob.words.count(word) / len(blob.words)
def n_containing(word, bloblist):
return sum(1 for blob in bloblist if word in blob)
def idf(word, bloblist):
return math.log(len(bloblist) / (1 + n_containing(word, bloblist)))
def tfidf(word, blob, bloblist):
return tf(word, blob) * idf(word, bloblist)
document1 = tb("""Today, the weather is 30 degrees in Celcius. It is really hot""")
document2 = tb("""I can't believe the traffic headed to the beach. It is really a circus out there.'""")
document3 = tb("""There are so many tolls on this road. I recommend taking the interstate.""")
bloblist = [document1, document2, document3]
for i, blob in enumerate(bloblist):
print("Document {}".format(i + 1))
scores = {word: tfidf(word, blob, bloblist) for word in blob.words}
sorted_words = sorted(scores.items(), key=lambda x: x[1], reverse=True)
for word, score in sorted_words:
score_weight = score * 100
print("\t{}, {}".format(word, round(score_weight, 5)))
</code></pre>
<p>我想在一个目录中使用一个输入txt文件,而不是每个硬编码的<code>document</code>。在</p>
<p>例如,假设我有一个目录<code>foo</code>,它包含三个文件<code>file1</code>,<code>file2</code>,<code>file3</code>。在</p>
<p>文件1包含<code>document1</code>包含的内容,即</p>
<p>文件1:</p>
^{pr2}$
<p>文件2包含<code>document2</code>包含的内容,即</p>
<pre><code>I can't believe the traffic headed to the beach. It is really a circus out there.
</code></pre>
<p>文件3包含<code>document3</code>包含的内容,即</p>
<pre><code>There are so many tolls on this road. I recommend taking the interstate.
</code></pre>
<p>我不得不使用<code>glob</code>来实现我想要的结果,我提出了以下代码适配器,它正确地标识了文件,但不像原始代码那样单独处理它们:</p>
<pre><code>file_names = glob.glob("/path/to/foo/*")
files = map(open,file_names)
documents = [file.read() for file in files]
[file.close() for file in files]
bloblist = [documents]
for i, blob in enumerate(bloblist):
print("Document {}".format(i + 1))
scores = {word: tfidf(word, blob, bloblist) for word in blob.words}
sorted_words = sorted(scores.items(), key=lambda x: x[1], reverse=True)
for word, score in sorted_words:
score_weight = score * 100
print("\t{}, {}".format(word, round(score_weight, 5)))
</code></pre>
<p>如何使用<code>glob</code>维护每个文件的分数?在</p>
<p>在使用目录中的文件作为输入后,所需的结果将与原始代码相同[空间的结果排到前3位]:</p>
<pre><code>Document 1
Celcius, 3.37888
30, 3.37888
hot, 3.37888
Document 2
there, 2.38509
out, 2.38509
headed, 2.38509
Document 3
on, 3.11896
this, 3.11896
many, 3.11896
</code></pre>
<p>类似的问题<a href="https://stackoverflow.com/questions/22434092/compute-tf-idf-with-corpus">here</a>没有完全解决问题。我想知道如何调用这些文件来计算<code>idf</code>,但要分别维护它们来计算完整的<code>tf-idf</code>?在</p>