检查从twi中提取的趋势的语言

Question

我只是使用python中的tweepy模块从twitter中提取顶级hashtag。有一个我面临的主要问题，我想检查一下标签是否是英文的。非英语标签应删除。你知道吗

示例：

tags=['AskOrange','CharlestonShooting','ReplyToASong','UberLIVE','Otecmatkasyn']

不应该有Otecmatkasyn。你知道吗

Answer 1

import re,urllib2
def find_words(each_func):
    i=0
    wordsineach_func=[] 
    while len(each_func) >0:
        i=i+1
        word_found=longest_word(each_func)
        if len(word_found)>0:
            wordsineach_func.append(word_found)
            each_func=each_func.replace(word_found,"") 
     #   print i,word_found,each  
    return wordsineach_func       

def longest_word(phrase):
    phrase_length=len(phrase)
    words_found=[];index=0
    outerstring=""
    while index < phrase_length:
        outerstring=outerstring+phrase[index]
        index=index+1
        if outerstring in words or outerstring.lower() in words:
            words_found.append(outerstring)
    if len(words_found) ==0:
        words_found.append(phrase)
    return max(words_found, key=len)

data = urllib2.urlopen('https://s3.amazonaws.com/hr-testcases/479/assets/words.txt')
words=[]
for line in data:
    words.append(line.replace("\n",""))
string="#honesthournow20"
string=string.replace("#","")
new_words=re.split(r'(\d+)',string)
output=[]
for each in new_words:
    each_words=find_words(each)
    for each_word in each_words:
        output.append(each_word)
print output

然后检查语言。你知道吗

Answer 2

您需要使用的是语言检测器API。好的是由Google提供的，但不是免费的。另一个好的选择是Language Detection API。你知道吗

在为自己选择了最好的API之后，您需要解析文本，使其成为一个句子。例如，标记'AskOrange'必须拆分为'Ask Orange'。您可以迭代字符串的每个字符，检查其是否为大写，并在其中插入空格：

new_tags = []
for tag in tags:
    new_word = tag
    uppercases = 0 # In case your sentence has several uppercases
    for i in xrange(1, len(tag)):
        if tag[i].istitle():
            new_word = new_word[:i+uppercases] + ' ' + new_word[i+uppercases:]
            uppercases = uppercases + 1
    new_tags.append(new_word)

最后，将new_tags列表发送到API以检测语言。你知道吗