擅长:python、mysql、java
<p>在scrapy 0.19.x中,您应该执行以下操作:</p>
<pre><code>from twisted.internet import reactor
from scrapy.crawler import Crawler
from scrapy import log, signals
from testspiders.spiders.followall import FollowAllSpider
from scrapy.utils.project import get_project_settings
spider = FollowAllSpider(domain='scrapinghub.com')
settings = get_project_settings()
crawler = Crawler(settings)
crawler.signals.connect(reactor.stop, signal=signals.spider_closed)
crawler.configure()
crawler.crawl(spider)
crawler.start()
log.start()
reactor.run() # the script will block here until the spider_closed signal was sent
</code></pre>
<p>注意这些行</p>
<pre><code>settings = get_project_settings()
crawler = Crawler(settings)
</code></pre>
<p>没有它,你的蜘蛛将不会使用你的设置,也不会保存项目。
我花了一段时间才弄明白为什么文档中的示例没有保存我的项目。我发送了一个请求来修复doc示例。</p>
<p>另一种方法是直接从脚本调用命令</p>
<pre><code>from scrapy import cmdline
cmdline.execute("scrapy crawl followall".split()) #followall is the spider's name
</code></pre>