Scrapy如何修复已启用Cookie时的“请启用Cookie”?

2024-09-30 04:29:31 发布

您现在位置:Python中文网/ 问答频道 /正文

我正在尝试为一个大学项目的鞋爬行一个网站。从下面的代码中,我得到了3-4只鞋,但是在这之后,对于剩余的鞋,我得到了响应“请启用cookies”,尽管我设置了Cookies_enabled=TrueCookies_debug=True。问题在哪里

import scrapy
from scrapy.http import Request


class FirespiderSpider(scrapy.Spider):
    name = 'firespider'
    page_number = 2
    allowed_domains = ['kicksonfire.com']
    start_urls = ['https://www.kicksonfire.com/app/upcoming?page=1']

    def parse(self, response):
        shoe_links = response.xpath ('//div/a/@href').extract()[1:None]
        for link in shoe_links:
            yield scrapy.Request(link, callback=self.parse_page, dont_filter=True)


    def parse_page(self, response):
        shoe_name = response.xpath('//div/h2/text()').extract()[0].strip()
        print('Schuh:' + shoe_name)

First part of the logResponse "Please enable cookies" in the log


Tags: nameimportselfcomtrueparseresponserequest

热门问题