我正试图刮一个网页与钉住,但呼吁回不分析的项目,任何帮助将不胜感激…这里是代码
# -*- coding: utf-8 -*-
import scrapy
from ..items import EscrotsItem
class Escorts(scrapy.Spider):
name = 'escorts'
allowed_domains = ['www.escortsandbabes.com.au']
start_urls = ['https://escortsandbabes.com.au/Directory/ACT/Canberra/2600/Any/All/']
def parse_links(self, response):
for i in response.css('.btn.btn-default.btn-block::attr(href)').extract()[2:]:
yield scrapy.Request(url=response.urljoin(i),callback=self.parse)
NextPage = response.css('.page.next-page::attr(href)').extract_first()
if NextPage:
yield scrapy.Request(
url=response.urljoin(NextPage),
callback=self.parse_links)
def parse(self, response):
for x in response.xpath('//div[@class="advertiser-profile"]'):
item = EscrotsItem()
item['Name'] = x.css('.advertiser-names--display-name::text').extract_first()
item['Username'] = x.css('.advertiser-names--username::text').extract_first()
item['Phone'] = x.css('.contact-number::text').extract_first()
yield item
您的代码从
start_urls
调用url并转到parse
函数。因为没有任何div.advertiser-profile
元素,所以它确实应该在没有任何结果的情况下关闭。所以根本不调用parse_links
函数。你知道吗更改函数名称:
我来自scrapy shell的日志:
相关问题 更多 >
编程相关推荐