java Nutch爬网错误输入路径不存在
我有带有2个datanode服务器的nutch/hadoop。我尝试爬网一些URL,但nutch失败,出现以下错误:
Fetcher: segment: crawl/segments
Fetcher: org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: hdfs://devcluster01:9000/user/nutch/crawl/segments/crawl_generate
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:190)
at org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:44)
at org.apache.nutch.fetcher.Fetcher$InputFormat.getSplits(Fetcher.java:105)
at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:810)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:781)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1249)
at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:1107)
at org.apache.nutch.fetcher.Fetcher.run(Fetcher.java:1145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.nutch.fetcher.Fetcher.main(Fetcher.java:1116)
有人能帮我吗?我不知道怎么解决这个问题! 很多很多Thx
# 1 楼答案
验证nutch/crawl/segments/crawl_生成路径是否正确
路径错误或分析阶段未完成
# 2 楼答案
nutch的生成阶段在segments目录中创建“crawl_generate”。它包含在获取阶段使用的获取列表。出现的错误是因为获取阶段无法获取获取列表。确保generate的输出填充在fetch试图查找它的位置