擅长:python、mysql、java
<p>您的输出/日志中是否有这样一行:</p>
<pre><code>Expected response size X larger than download max size Y.
</code></pre>
<p>听起来您请求的响应超过1GB。您的错误来自于<a href="https://github.com/scrapy/scrapy/blob/1.0.5/scrapy/core/downloader/handlers/http11.py#L246" rel="nofollow">download handler</a>,它<a href="http://doc.scrapy.org/en/latest/topics/settings.html#download-maxsize" rel="nofollow">defaults to one gig</a>,但是可以很容易地<a href="http://doc.scrapy.org/en/latest/topics/settings.html#populating-the-settings" rel="nofollow">overridden</a>:</p>
<ul>
<li>带有<a href="http://doc.scrapy.org/en/latest/topics/settings.html#download-maxsize" rel="nofollow">DOWNLOAD_MAXSIZE</a>的项目<code>settings.py</code>文件</li>
<li>蜘蛛的<a href="http://doc.scrapy.org/en/latest/topics/spiders.html#scrapy.spiders.Spider.custom_settings" rel="nofollow">^{<cd2>}</a>与{<cd3>}或</li>
<li>一本<code>download_maxsize</code>的手册<a href="http://doc.scrapy.org/en/latest/topics/request-response.html#scrapy.http.Request.meta" rel="nofollow">Request meta key</a>。在</li>
</ul>