程序读取多个网址，并生成一个字（应该只包含字母数字）频率标签

Question

我正在编写一个Python程序来读取多个url并生成一个单词（一个单词只包含字母a-Za-z0-9）频率表。输出可以存储在名为url1.txt、url2.txt的文件中

到目前为止，我的情况是：

import urllib2
import obo

url = 'sample url'

response = urllib2.urlopen(url)
html = response.read()
text = obo.stripTags(html).lower()
wordlist = obo.stripNonAlphaNum(text)


for s in sorteddict: 
    print str(s)

Answer 1

您可以使用boilerpipe轻松地提取文本：https://github.com/misja/python-boilerpipe。你知道吗

代码可能如下所示：

from boilerpipe.extract import Extractor
from collections import Counter

urls = ['url1', 'url2', ... ] # A list of the urls you want to fetch
# Ask boilerpipe to fetch the data
extractors = [Extractor(extractor='ArticleExtractor', url=url) for url in urls]
# Ask boilerpipe to extract the text
raw_texts = [extractor.getText() for extractor in extractors]
# count the occurrences of words in each text
word_counts = [Counter(text.split(" ")) for text in raw_texts]