下面是我编写的一个爬行器,用于爬网RSS提要并提取第一个链接和图像标题,然后将它们保存到文本文件中:
import scrapy
class artSpider(scrapy.Spider):
name = "metart"
start_urls = ['https://www.metmuseum.org/art/artwork-of-the-day?rss=1']
def parse(self, response):
title = response.xpath('//item/title/text()').extract_first().replace(" ", "_")
description = response.xpath('//item/description/text()').extract_first()
lnkstart = description.find("https://image")
lnkcut1 = description.find("web-highlight")
lnkcut2 = lnkcut1 + 13
lnkend = description.find(".jpg") + 4
link = description[lnkstart:lnkcut1] + "original" + description[lnkcut2:lnkend]
ttlstart = description.find("who=") + 4
ttlend = description.find("&rss=1")
filename = "/path/to/save/folder/" + description[ttlstart:ttlend].replace("+", "_") + "-" + title + ".jpg"
print(filename)
print(link)
filename_file = open('filename_metart.txt', 'w')
filename_file.write(filename)
filename_file.close
link_file = open('link_metart.txt', 'w')
link_file.write(link)
link_file.close
它浏览了大都会博物馆的“当日艺术品”RSS提要,找到了最新的艺术品。然后解析标题和原始链接(RSS提要中的链接用于缩略图),并将它们保存到单个文本文件中。标题用于在下载图像时生成文件名。你知道吗
我知道解析函数是一团糟,我保存链接和文件名的方式很难看。我刚开始与Scrapy,只是想让蜘蛛的工作,因为它只是一个更广泛的项目,我正在工作的一部分。你知道吗
该项目本身从artwork/image/PhotoofDay提要和网站中提取图像,下载图像并将其设置为桌面背景幻灯片。你知道吗
我的问题是当我启动蜘蛛时,它会返回一个“NoneType”的响应。但是如果我在浏览器(https://metmuseum.org/art/artwork-of-the-day?rss=1)中转到RSS提要,然后运行spider,它就会正常工作。你知道吗
失败的进程
scrapy crawl metart
title
的类型为NoneType
有效的流程
scrapy crawl metart
我已经做了一些故障排除
我用了一个粗糙的壳来复制蜘蛛所经历的过程,效果很好。我从蜘蛛走过的每一步开始:
fetch("https://metmuseum.org/art/artwork-of-the-day?rss=1")
然后输入parse函数的每一行。这样可以很好地工作,并且可以保存正确的链接和文件名,而不必首先在web浏览器中打开URL。你知道吗
为了完整,下面是我的剪贴画项目设置.py文件:
BOT_NAME = 'wallpaper_scraper'
SPIDER_MODULES = ['wallpaper_scraper.spiders']
NEWSPIDER_MODULE = 'wallpaper_scraper.spiders'
# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = 'cchowgule for a wallpaper slideshow'
# Obey robots.txt rules
ROBOTSTXT_OBEY = True
我很困惑,在网络浏览器中打开URL的行为会如何影响蜘蛛的响应。你知道吗
任何帮助清理我的解析函数也将是可爱的。你知道吗
谢谢
更新
在协调世界时05时35分,我运行scrapy crawl metart
,得到以下响应:
2018-06-23 11:02:54 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: wallpaper_scraper)
2018-06-23 11:02:54 [scrapy.utils.log] INFO: Versions: lxml 4.2.1.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.4.0, w3lib 1.19.0, Twisted 17.9.0, Python 2.7.15 (default, May 1 2018, 05:55:50) - [GCC 7.3.0], pyOpenSSL 17.5.0 (OpenSSL 1.1.0h 27 Mar 2018), cryptography 2.2.2, Platform Linux-4.16.0-2-amd64-x86_64-with-debian-buster-sid
2018-06-23 11:02:54 [scrapy.crawler] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'wallpaper_scraper.spiders', 'SPIDER_MODULES': ['wallpaper_scraper.spiders'], 'ROBOTSTXT_OBEY': True, 'USER_AGENT': 'cchowgule for a wallpaper slideshow', 'BOT_NAME': 'wallpaper_scraper'}
2018-06-23 11:02:54 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.memusage.MemoryUsage',
'scrapy.extensions.logstats.LogStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.corestats.CoreStats']
2018-06-23 11:02:54 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2018-06-23 11:02:54 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2018-06-23 11:02:54 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2018-06-23 11:02:54 [scrapy.core.engine] INFO: Spider opened
2018-06-23 11:02:54 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-06-23 11:02:54 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-06-23 11:02:56 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.metmuseum.org/robots.txt> (referer: None)
2018-06-23 11:02:56 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.metmuseum.org/art/artwork-of-the-day?rss=1> (referer: None)
2018-06-23 11:02:56 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.metmuseum.org/art/artwork-of-the-day?rss=1> (referer: None)
Traceback (most recent call last):
File "/home/cchowgule/.local/lib/python2.7/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/home/cchowgule/WD/pyenvscrapy/wallpaper_scraper/wallpaper_scraper/spiders/metart.py", line 9, in parse
title = response.xpath('//item/title/text()').extract_first().replace(" ", "_")
AttributeError: 'NoneType' object has no attribute 'replace'
2018-06-23 11:02:57 [scrapy.core.engine] INFO: Closing spider (finished)
2018-06-23 11:02:57 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 646,
'downloader/request_count': 2,
'downloader/request_method_count/GET': 2,
'downloader/response_bytes': 1725,
'downloader/response_count': 2,
'downloader/response_status_count/200': 2,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2018, 6, 23, 5, 32, 57, 65104),
'log_count/DEBUG': 3,
'log_count/ERROR': 1,
'log_count/INFO': 7,
'memusage/max': 52793344,
'memusage/startup': 52793344,
'response_received_count': 2,
'scheduler/dequeued': 1,
'scheduler/dequeued/memory': 1,
'scheduler/enqueued': 1,
'scheduler/enqueued/memory': 1,
'spider_exceptions/AttributeError': 1,
'start_time': datetime.datetime(2018, 6, 23, 5, 32, 54, 464466)}
2018-06-23 11:02:57 [scrapy.core.engine] INFO: Spider closed (finished)
我用同样的结果运行了scrapy crawl metart
5次。你知道吗
然后我打开一个浏览器,转到https://metmuseum.org/art/artwork-of-the-day?rss=1,再次运行scrapy crawl metart
。这次它工作正常。。。。。。我就是不明白。你知道吗
目前没有回答
相关问题 更多 >
编程相关推荐