Scrapy：在重定向之前获取URL的特定部分

Question

下面是我将使用的代码（我正在使用scrapy）

def start_requests(self):
        start_urls = ['https://www.lowes.com/search?searchTerm=8654RM-42']

这是我存储所有URL的地方

---

下面是我如何尝试只打印“=”之后的所有内容

            productSKU = response.url.split("=")[-1]
            item["productSKU"] = productSKU

---

以下是输出：

{'productPrice': '1,449.95',
 'productSKU': 'https://www.lowes.com/pd/ZLINE-KITCHEN-BATH-Ducted-Red-Matte-Wall-Mounted-Range-Hood-Common-42-Inch-Actual-42-in/1001440644'}

---

现在问题来了：

我输入的URL最终将被填充

https://www.lowes.com/search?searchTerm = {something}

这就是为什么我想使用{something}来确保我将在CSV上获取我试图获取的所有项目（用于排序和匹配目的）

我正在使用的URL将重定向到此URL：

（输入）https://www.lowes.com/search?searchTerm=8654RM-42

->

（重定向）https://www.lowes.com/pd/ZLINE-KITCHEN-BATH-Ducted-Red-Matte-Wall-Mounted-Range-Hood-Common-42-Inch-Actual-42-in/1001440644

因此，我对productSKU的输出是整个重定向URL，而不仅仅是“=”符号后面的内容。我想要的输出是8654RM-42

这是我的整个计划

# -*- coding: utf-8 -*-
import scrapy
from ..items import LowesspiderItem
from scrapy.http import Request

class LowesSpider(scrapy.Spider):
name = 'lowes'

def start_requests(self):
    start_urls = ['https://www.lowes.com/search?searchTerm=8654RM-42']

    for url in start_urls:
        yield Request(url, cookies={'sn':'2333'}) #Added cookie to bypass location req 

def parse(self, response):
    items = response.css('.grid-container')
    for product in items:
        item = LowesspiderItem()

    #get product price
        productPrice = product.css('.art-pd-price::text').get()
        productSKU = response.url.split("=")[-1]


        item["productSKU"] = productSKU
        item["productPrice"] = productPrice


        yield item

Answer 1

您需要使用meta像这样传入输入url

def start_requests(self):
    start_urls = ['https://www.lowes.com/search?searchTerm=8654RM-42']

    for url in start_urls:
        yield Request(url, cookies={'sn':'2333'},meta={'url':url)
def parse(self,response):
    url = response.meta['url'] #your input url