我想为https://www.trekearth.com启动scrapy shell 之后
scrapy shell https://www.trekearth.com
我收到了
2018-05-11 16:02:04 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying
<GET https://www.trekearth.com> (failed 1 times): 524 Unknown Status
2018-05-11 16:02:05 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying
<GET https://www.trekearth.com> (failed 2 times): 502 Bad Gateway
2018-05-11 16:03:45 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up
retrying <GET https://www.trekearth.com> (failed 3 times): 524 Unknown Status
原因是什么?我查过的所有其他网站都不会返回类似的结果。你知道吗
它是一种按用户代理的过滤器:
相关问题 更多 >
编程相关推荐