当我已经解析了一个页面并从中提取信息时,我正在尝试跟踪Scrapy中的链接。问题是,这个网页没有href,所以我不能简单地跟随它。我用@data param扩展了XPath查询,最后得到了一个结果:page=2
问题是我不确定如何遵循这个链接,因为我想将listName["listLinkMaker"]
传递给我的URL生成器或编写器
我是否应该制作另一个“def”并称之为def parse_pagination以跟踪链接
代码中使用的JSON非常简单:
[
{"storeName": "Interspar", "storeLinkMaker": "https://popusti.njuskalo.hr/trgovina/Interspar"}
]
代码如下:
# -*- coding: utf-8 -*-
import scrapy
import json
class LoclocatorTestSpider(scrapy.Spider):
name = "loclocator_test"
start_urls = []
with open("test_one_url.json", encoding="utf-8") as json_file:
data = json.load(json_file)
for store in data:
storeName = store["storeName"]
storeLinkUrl = store["storeLinkMaker"]
start_urls.append(storeLinkUrl)
def parse(self, response):
selector = "//div[@class='mainContentWrapInner cf']"
store_name_selector = ".//h1[@class='title']/text()"
store_branches_selector = ".//li/a[@class='xiti']/@href"
for basic_info in response.xpath(selector):
store_branches = {}
store_branches["storeName"] = basic_info.xpath(store_name_selector).extract_first()
# This specific XPath extracts 1st part of link needed to crawl all of store branches
store_branches["storeBranchesLink"] = basic_info.xpath(store_branches_selector).extract_first() + "?"
store_branches_url = basic_info.xpath(store_branches_selector).extract_first()
yield response.follow(store_branches_url, self.parse_pagination, meta={"store_branches": store_branches})
def parse_branches(self, response):
store_branches_name_selector = "//li[@class='xiti']"
store_branches = response.meta["store_branches"]
for store_branch in response.xpath(store_branches_name_selector):
store_branches["storeBranchName"] = store_branch.xpath(".//span[@class='title']/text()").extract_first()
yield store_branches
# This specific XPath extracts 2nd part of link needed to crawl all of store branches
# URL should look like: https://popusti.njuskalo.hr/trgovina/Interspar?page=n where n>0
links = response.selector.xpath("//li[@class='next']/button[@class='nBtn link xiti']/@data-param").extract()
for link in links:
absolute_url = #LIST FROM FIRST PARSE (ie. store_branches["storeBranchesLink"]) + link
yield scrapy.Request(absolute_url, callback=self.parse_branches)
多谢各位
我自己设法找到了一个解决方案,而且我离这个解决方案比较近
在该部分下:
我认为解决方案是添加store_分支的响应,因为它能够找到所有可能的页面(?page=n,其中n>;0)。如果有人知道更多的技术信息,因为我对代码的理解还比较初级,请务必回答
相关问题 更多 >
编程相关推荐