Scrapy调试：重定向301以获取

Question

在写了我的第一个？递归？蜘蛛，我面临一些问题，我一整天都无法解决

我做过研究，但我尝试过的每一个解决方案都没有帮到我

我的控制台输出

我的修改设置.py

USER_AGENT = 'kartonage (Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:31.0) Gecko/20100101 Firefox/31.0)'
DOWNLOAD_DELAY = 0.5
HTTPERROR_ALLOW_ALL = True

这个user_代理和httperror_allow_都是针对具有重定向301错误的其他人的一些解决方案

My modified items.py

import scrapy


class KartonageItem(scrapy.Item):
    SKU = scrapy.Field()
    Title = scrapy.Field()
    Link = scrapy.Field()
    Price = scrapy.Field()
    Delivery_Status = scrapy.Field()
    Weight = scrapy.Field()
    QTY = scrapy.Field()
    Volume = scrapy.Field()

我使用的代码

import scrapy
from ..items import KartonageItem

class KartonSpider(scrapy.Spider):
    name = "kartons12"
    allow_domains = ['karton.eu']
    start_urls = [
        'https://www.karton.eu/Faltkartons'
        ]
    custom_settings = {'FEED_EXPORT_FIELDS': ['SKU', 'Title', 'Link', 'Price', 'Delivery_Status', 'Weight', 'QTY', 'Volume'] } 
    
    def parse(self, response):
        url = response.xpath('//div[@class="cat-thumbnails"]')

        for a in url:
            link = a.xpath('a/@href')
            yield response.follow(url=link.get(), callback=self.parse_category_cartons)

    def parse_category_cartons(self, response):
        url2 = response.xpath('//div[@class="cat-thumbnails"]')

        for a in url2:
            link = a.xpath('a/@href')
            yield response.follow(url=link.get(), callback=self.parse_target_page)

    def parse_target_page(self, response):
        card = response.xpath('//div[@class="text-center articelbox"]')

        for a in card:
            items = KartonageItem()
            link = a.xpath('a/@href')
            items ['SKU'] = a.xpath('.//div[@class="delivery-status"]/small/text()').get()
            items ['Title'] = a.xpath('.//h5[@class="title"]/a/text()').get()
            items ['Link'] = a.xpath('.//h5[@class="text-center artikelbox"]/a/@href').extract()
            items ['Price'] = a.xpath('.//strong[@class="price-ger price text-nowrap"]/span/text()').get()
            items ['Delivery_Status'] = a.xpath('.//div[@class="signal_image status-2"]/small/text()').get()
            yield response.follow(url=link.get(),callback=self.parse_item, meta={'items':items})

    def parse_item(self,response):
        table = response.xpath('//span[@class="product-info-inner"]')

        items = KartonageItem()
        items = response.meta['items']
        items['Weight'] = a.xpath('.//span[@class="staffelpreise-small"]/text()').get()
        items['Volume'] = a.xpath('.//td[@class="icon_contenct"][7]/text()').get()
        yield items

Answer 1

HTTP 301不是错误，它是对Moved Permanently的响应。它会自动将您重定向到该页面的新地址。您可以在执行日志中看到已重定向

这本身不应该是个问题。这可能是其他原因造成的吗？蜘蛛有什么意外的行为吗