我用pythonlucene索引url页面。在
我在尝试向文档添加字段时出错。我不知道为什么。 错误说明:
Java错误:,>; Java堆栈跟踪: java.lang.IllegalArgumentException:既没有索引也没有存储的字段没有意义 在org.apache.lucene.文档.字段.(Field.java:249)在
按照我的说法:文件添加(字段(“内容”,文本,t2))
我使用的python代码是:
def IndexerForUrl(start, number, domain):
lucene.initVM()
# join base dir and index dir
path = os.path.abspath("paths")
directory = SimpleFSDirectory(Paths.get(path)) # the index
analyzer = StandardAnalyzer()
writerConfig = IndexWriterConfig(analyzer)
writerConfig.setOpenMode(IndexWriterConfig.OpenMode.CREATE)
writer = IndexWriter(directory, writerConfig)
print "reading lines from sys.std..."
# hashtable dictionary
D = {}
D[start] = [start]
numVisited = 0
wordBool = False
n = start
queue = [start]
visited = set()
t1 = FieldType()
t1.setStored(True)
t1.setTokenized(False)
t2 = FieldType()
t2.setStored(False)
t2.setTokenized(True)
while numVisited < number and queue and not wordBool:
pg = queue.pop(0)
if pg not in visited:
visited.add(pg)
htmlwebpg = urllib2.urlopen(pg).read()
# robot exclusion standard
rp = robotparser.RobotFileParser()
rp.set_url(pg)
rp.read() # read robots.txt url and feeds to parser
soup = BeautifulSoup(htmlwebpg, 'html.parser')
for script in soup(["script","style"]):
script.extract()
text = soup.get_text()
lines = (line.strip() for line in text.splitlines())
chunks = (phrase.strip() for line in lines for phrase in line.split(" "))
text = '\n'.join(chunk for chunk in chunks if chunk)
print text
doc = Document()
doc.add(Field("urlpath", pg, t2))
if len(text)> 0:
doc.add(Field("contents", text, t2))
else:
print "warning: no content in %s " % pgv
writer.addDocument(doc)
numVisited = numVisited+1
linkset = set()
# add to list
for link in soup.findAll('a', attrs={'href':re.compile("^http://")}):
#links.append(link.get('href'))
if rp.can_fetch(link.get('href')):
linkset.add(link.get('href'))
D[pg] = linkset
queue.extend(D[pg] - visited)
writer.commit()
writer.close()
directory.close() #close the index
return writer
如果一个字段既没有被索引也没有被存储,它将不会以任何方式在索引中表示,因此它在那里没有意义。我猜你想索引字段类型t2。为此,您需要set the IndexOptions,类似于:
相关问题 更多 >
编程相关推荐