如何为斯坦福德纳清理句子

Question

我想在python中使用StanfordNER来检测命名实体。我该如何整理句子？在

例如，考虑

qry="In the UK, the class is relatively crowded with Zacc competing with Abc's Popol (market leader) and Xyz's Abcvd."

如果我这么做

st = StanfordNERTagger('english.all.3class.distsim.crf.ser.gz') 
print st.tag(qry.split())

我明白了

^{pr2}$

`

因此只检测到1个命名实体。但是，如果我清除了所有的字符，用一些特殊的空格替换

qry="In the UK the class is relatively crowded with Zacc competing with Abc s Popol market leader and Xyz s Abcvd"

我明白了

[
    (u'In', u'O'), (u'the', u'O'), (u'UK', u'LOCATION'), (u'the', u'O'), 
    (u'class', u'O'), (u'is', u'O'), (u'relatively', u'O'), (u'crowded', u'O'), 
    (u'with', u'O'), (u'Zacc', u'PERSON'), (u'competing', u'O'), (u'with', u'O'), 
    (u'Abc', u'ORGANIZATION'), (u's', u'O'), (u'Popol', u'PERSON'), (u'market', u'O'), 
    (u'leader', u'O'), (u'and', u'O'), (u'Xyz', u'ORGANIZATION'), (u's', u'O'), (u'Abcvd', u'PERSON')]

`

所以很明显，这更合适。关于如何清理StanfordNER的句子有什么一般规则吗？一开始我以为根本不需要清理！在

Answer 1

在处理文本之前，请用word标记文本。另外，请注意，大多数注释系统都是从句子中训练出来的，所以您可以在单词标记化之前进行句子标记化。在

alvas@ubi:~$ export STANFORDTOOLSDIR=$HOME
alvas@ubi:~$ export CLASSPATH=$STANFORDTOOLSDIR/stanford-ner-2015-12-09/stanford-ner.jar
alvas@ubi:~$ export STANFORD_MODELS=$STANFORDTOOLSDIR/stanford-ner-2015-12-09/classifiers
alvas@ubi:~$ python
Python 2.7.11 (default, Dec 15 2015, 16:46:19) 
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from nltk import word_tokenize
>>> from nltk.tag import StanfordNERTagger
>>> from nltk.internals import find_jars_within_path
>>> st = StanfordNERTagger('english.all.3class.distsim.crf.ser.gz')
>>> stanford_dir = st._stanford_jar.rpartition('/')[0]
>>> stanford_jars = find_jars_within_path(stanford_dir)
>>> st._stanford_jar = ':'.join(stanford_jars)
>>> 
>>> text = "In the UK, the class is relatively crowded with Zacc competing with Abc's Popol (market leader) and  Xyz's Abcvd."
>>> text = word_tokenize(text)
>>> text
['In', 'the', 'UK', ',', 'the', 'class', 'is', 'relatively', 'crowded', 'with', 'Zacc', 'competing', 'with', 'Abc', "'s", 'Popol', '(', 'market', 'leader', ')', 'and', 'Xyz', "'s", 'Abcvd', '.']
>>> st.tag(text)
[(u'In', u'O'), (u'the', u'O'), (u'UK', u'LOCATION'), (u',', u'O'), (u'the', u'O'), (u'class', u'O'), (u'is', u'O'), (u'relatively', u'O'), (u'crowded', u'O'), (u'with', u'O'), (u'Zacc', u'PERSON'), (u'competing', u'O'), (u'with', u'O'), (u'Abc', u'PERSON'), (u"'s", u'O'), (u'Popol', u'O'), (u'(', u'O'), (u'market', u'O'), (u'leader', u'O'), (u')', u'O'), (u'and', u'O'), (u'Xyz', u'ORGANIZATION'), (u"'s", u'O'), (u'Abcvd', u'O'), (u'.', u'O')]

Answer 2

你可以使用斯坦福标记器。 你可以使用下面的代码。在

from nltk.tokenize.stanford import StanfordTokenizer
token = StanfordTokenizer('stanford-ner-2014-06-16/stanford-ner.jar')
qry="In the UK, the class is relatively crowded with Zacc competing with Abc's Popol (market leader) and  Xyz's Abcvd."
tok = token.tokenize(qry)
print tok

你将得到你需要的代币。在

[u'In',
u'the',
u'UK',
u',',
u'the',
u'class',
u'is',
u'relatively',
u'crowded',
u'with',
u'Zacc',
u'competing',
u'with',
u'Abc',
u"'s",
u'Popol',
u'-LRB-',
u'market',
u'leader',
u'-RRB-',
u'and',
u'Xyz',
u"'s",
u'Abcvd',
u'.']

Answer 3

你应该确保你正在标记这个句子这是第一次调用（你用qry.split()隐式标记错误）和第二次调用（例如，posessive 's作为它自己的标记）之间的巨大区别。Stanforddoes have a tokenizer，这是NER系统的标记器，尽管我不是如何从Python调用它的专家。简单地把句子分开就可以给你做标记吗？在