在浏览网页后,粗糙的响应会发生变化

2024-10-03 02:42:04 发布

您现在位置:Python中文网/ 问答频道 /正文

下面是我编写的一个爬行器,用于爬网RSS提要并提取第一个链接和图像标题,然后将它们保存到文本文件中:

import scrapy

class artSpider(scrapy.Spider):
    name = "metart"

    start_urls = ['https://www.metmuseum.org/art/artwork-of-the-day?rss=1']

    def parse(self, response):
        title = response.xpath('//item/title/text()').extract_first().replace(" ", "_")
        description = response.xpath('//item/description/text()').extract_first()

        lnkstart = description.find("https://image")
        lnkcut1 = description.find("web-highlight")
        lnkcut2 = lnkcut1 + 13
        lnkend = description.find(".jpg") + 4

        link = description[lnkstart:lnkcut1] + "original" + description[lnkcut2:lnkend]

        ttlstart = description.find("who=") + 4
        ttlend = description.find("&rss=1")

        filename = "/path/to/save/folder/" + description[ttlstart:ttlend].replace("+", "_") + "-" + title + ".jpg"

        print(filename)
        print(link)

        filename_file = open('filename_metart.txt', 'w')
        filename_file.write(filename)
        filename_file.close
        link_file = open('link_metart.txt', 'w')
        link_file.write(link)
        link_file.close

它浏览了大都会博物馆的“当日艺术品”RSS提要,找到了最新的艺术品。然后解析标题和原始链接(RSS提要中的链接用于缩略图),并将它们保存到单个文本文件中。标题用于在下载图像时生成文件名。你知道吗

我知道解析函数是一团糟,我保存链接和文件名的方式很难看。我刚开始与Scrapy,只是想让蜘蛛的工作,因为它只是一个更广泛的项目,我正在工作的一部分。你知道吗

该项目本身从artwork/image/PhotoofDay提要和网站中提取图像,下载图像并将其设置为桌面背景幻灯片。你知道吗

我的问题是当我启动蜘蛛时,它会返回一个“NoneType”的响应。但是如果我在浏览器(https://metmuseum.org/art/artwork-of-the-day?rss=1)中转到RSS提要,然后运行spider,它就会正常工作。你知道吗

失败的进程

  1. 呼叫scrapy crawl metart
    • 输出显示变量title的类型为NoneType
  2. 重复步骤1会得到相同的输出

有效的流程

  1. 在web浏览器中打开https://metmuseum.org/art/artwork-of-the-day?rss=1
  2. 呼叫scrapy crawl metart
    • 成功地将链接和文件名保存到相关文本文件

我已经做了一些故障排除

我用了一个粗糙的壳来复制蜘蛛所经历的过程,效果很好。我从蜘蛛走过的每一步开始:

fetch("https://metmuseum.org/art/artwork-of-the-day?rss=1")

然后输入parse函数的每一行。这样可以很好地工作,并且可以保存正确的链接和文件名,而不必首先在web浏览器中打开URL。你知道吗

为了完整,下面是我的剪贴画项目设置.py文件:

BOT_NAME = 'wallpaper_scraper'

SPIDER_MODULES = ['wallpaper_scraper.spiders']
NEWSPIDER_MODULE = 'wallpaper_scraper.spiders'

# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = 'cchowgule for a wallpaper slideshow'

# Obey robots.txt rules
ROBOTSTXT_OBEY = True

我很困惑,在网络浏览器中打开URL的行为会如何影响蜘蛛的响应。你知道吗

任何帮助清理我的解析函数也将是可爱的。你知道吗

谢谢

更新

在协调世界时05时35分,我运行scrapy crawl metart,得到以下响应:

2018-06-23 11:02:54 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: wallpaper_scraper)
2018-06-23 11:02:54 [scrapy.utils.log] INFO: Versions: lxml 4.2.1.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.4.0, w3lib 1.19.0, Twisted 17.9.0, Python 2.7.15 (default, May  1 2018, 05:55:50) - [GCC 7.3.0], pyOpenSSL 17.5.0 (OpenSSL 1.1.0h  27 Mar 2018), cryptography 2.2.2, Platform Linux-4.16.0-2-amd64-x86_64-with-debian-buster-sid
2018-06-23 11:02:54 [scrapy.crawler] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'wallpaper_scraper.spiders', 'SPIDER_MODULES': ['wallpaper_scraper.spiders'], 'ROBOTSTXT_OBEY': True, 'USER_AGENT': 'cchowgule for a wallpaper slideshow', 'BOT_NAME': 'wallpaper_scraper'}
2018-06-23 11:02:54 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.logstats.LogStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.corestats.CoreStats']
2018-06-23 11:02:54 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2018-06-23 11:02:54 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2018-06-23 11:02:54 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2018-06-23 11:02:54 [scrapy.core.engine] INFO: Spider opened
2018-06-23 11:02:54 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-06-23 11:02:54 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-06-23 11:02:56 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.metmuseum.org/robots.txt> (referer: None)
2018-06-23 11:02:56 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.metmuseum.org/art/artwork-of-the-day?rss=1> (referer: None)
2018-06-23 11:02:56 [scrapy.core.scraper] ERROR: Spider error processing <GET https://www.metmuseum.org/art/artwork-of-the-day?rss=1> (referer: None)
Traceback (most recent call last):
  File "/home/cchowgule/.local/lib/python2.7/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/cchowgule/WD/pyenvscrapy/wallpaper_scraper/wallpaper_scraper/spiders/metart.py", line 9, in parse
    title = response.xpath('//item/title/text()').extract_first().replace(" ", "_")
AttributeError: 'NoneType' object has no attribute 'replace'
2018-06-23 11:02:57 [scrapy.core.engine] INFO: Closing spider (finished)
2018-06-23 11:02:57 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 646,
 'downloader/request_count': 2,
 'downloader/request_method_count/GET': 2,
 'downloader/response_bytes': 1725,
 'downloader/response_count': 2,
 'downloader/response_status_count/200': 2,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2018, 6, 23, 5, 32, 57, 65104),
 'log_count/DEBUG': 3,
 'log_count/ERROR': 1,
 'log_count/INFO': 7,
 'memusage/max': 52793344,
 'memusage/startup': 52793344,
 'response_received_count': 2,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'spider_exceptions/AttributeError': 1,
 'start_time': datetime.datetime(2018, 6, 23, 5, 32, 54, 464466)}
2018-06-23 11:02:57 [scrapy.core.engine] INFO: Spider closed (finished)

我用同样的结果运行了scrapy crawl metart5次。你知道吗

然后我打开一个浏览器,转到https://metmuseum.org/art/artwork-of-the-day?rss=1,再次运行scrapy crawl metart。这次它工作正常。。。。。。我就是不明白。你知道吗


Tags: thehttpsorginforesponsecountdescriptionscraper