Scrapy旁路数据使用许可墙

Question

我正在使用下面的代码抓取雅虎财经新闻

class YfinNewsSpider(scrapy.Spider):
    name = 'yfin_news_spider'
    custom_settings = {'DOWNLOAD_DELAY': '0.5', 'COOKIES_ENABLED': True, 'COOKIES_DEBUG': True}

    def __init__(self, month, year, **kwargs):
        self.start_urls = ['https://finance.yahoo.com/sitemap/2020_03_all']
        self.allowed_domains = ['finance.yahoo.com']
        super().__init__(**kwargs)

    def parse(self, response):
        all_news_urls = response.xpath('//ul/li[@class="List(n) Py(3px) Lh(1.2)"]')
        for news in all_news_urls:
            news_url = news.xpath('.//a[@class="Td(n) Td(u):h C($c-fuji-grey-k)"]/@href').extract_first()
            yield scrapy.Request(news_url, callback=self.parse_news, dont_filter=True)

    def parse_news(self, response):
        news_url = str(response.url)
        title = response.xpath('//title/text()').extract_first()
        paragraphs = response.xpath('//div[@class="caas-body"]/p/text()').extract()
        date_time = response.xpath('//div[@class="caas-attr-time-style"]/time/@datetime').extract_first()

        yield {'title': title, 'url': news_url, 'body_text': paragraphs, 'timestamp': date_time}

然而，当我运行spider时，它会给出以下结果

2020-11-28 20:42:40 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://consent.yahoo.com/v2/collectConsent?sessionId=3_cc-session_05cc09ea-0bc0-439d-8b4c-2d6f20f52d6e> (referer: https://finance.yahoo.com/sitemap/2020_03_all)
2020-11-28 20:42:40 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <GET https://finance.yahoo.com/news/onegold-becomes-first-company-offer-110000241.html>
Cookie: B=cnmvgrdfs5a0r&b=3&s=o1; GUCS=ASXMbR9p
2020-11-28 20:42:40 [scrapy.core.scraper] DEBUG: Scraped from <200 https://consent.yahoo.com/v2/collectConsent?sessionId=3_cc-session_05cc09ea-0bc0-439d-8b4c-2d6f20f52d6e>
{'title': 'Yahoo er nu en del af Verizon Media', 'url': 'https://consent.yahoo.com/v2/collectConsent?sessionId=3_cc-session_05cc09ea-0bc0-439d-8b4c-2d6f20f52d6e', 'body_text': [], 'timestamp': None}
2020-11-28 20:42:41 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://consent.yahoo.com/v2/collectConsent?sessionId=3_cc-session_d6731ce6-78bc-4222-914f-24cf98f874b8> (referer: https://finance.yahoo.com/sitemap/2020_03_all)

这似乎表明，当我的spider转到https://finance.yahoo.com/sitemap/2020_03_all中找到的https://finance.yahoo.com/news/onegold-becomes-first-company-offer-110000241.html时。它试图将cookie发送到https://finance.yahoo.com/news/onegold-becomes-first-company-offer-110000241.html，但被重定向到https://consent.yahoo.com/v2/collectConsent?sessionId=3_cc-session_05cc09ea-0bc0-439d-8b4c-2d6f20f52d6e

我在浏览器中打开此同意墙https://consent.yahoo.com/v2/collectConsent?sessionId=3_cc-session_05cc09ea-0bc0-439d-8b4c-2d6f20f52d6e，发现数据同意接受屏幕。当我点击accept时，它将我带到了我想要刮取的正确站点。刮取结果也与此同意屏幕中的内容完全相同

我已尝试将COOKIES_ENABLED设置为True，但无效。那么，在scrapy中是否有绕过此接受屏幕的方法

多谢各位

Answer 1

你可以尝试一件事： 打开“网络”选项卡上的“同意”页面，然后单击“给予同意”按钮。在那里，您可以确定在您同意时它发送的请求。您可以尝试使用scrapy复制相同的请求。也许这样你的问题就会解决。 另一种选择是使用scrapy selenium手动单击该按钮，然后scrapy可以从那里接管