官方的小例子出错了？

Question

尝试了出现在documentation page上的示例 （名称下的示例：从单个回调返回多个请求和项）

我只是把域名改成了一个真正的网站：

import scrapy

class MySpider(scrapy.Spider):
    name = 'huffingtonpost'
    allowed_domains = ['huffingtonpost.com/']
    start_urls = [
        'http://www.huffingtonpost.com/politics/',
        'http://www.huffingtonpost.com/entertainment/',
        'http://www.huffingtonpost.com/media/',
    ]

    def parse(self, response):
        for h3 in response.xpath('//h3').extract():
            yield {"title": h3}

        for url in response.xpath('//a/@href').extract():
            yield scrapy.Request(url, callback=self.parse)

但是得到ValuError如在{a2}中发布的那样。 有什么想法吗？在

Answer 1

一些提取的链接是相对的（例如，/news/hillary-clinton/）。 你应该把它转换成绝对值（http://www.huffingtonpost.com/news/hillary-clinton/

import scrapy

class MySpider(scrapy.Spider):
    name = 'huffingtonpost'
    allowed_domains = ['huffingtonpost.com/']
    start_urls = [
        'http://www.huffingtonpost.com/politics/',
        'http://www.huffingtonpost.com/entertainment/',
        'http://www.huffingtonpost.com/media/',
    ]

    def parse(self, response):
        for h3 in response.xpath('//h3').extract():
            yield {"title": h3}

        for url in response.xpath('//a/@href').extract():
            if url.startswith('/'):
                # transform url into absolute
                url = 'http://www.huffingtonpost.com' + url
            if url.startswith('#'):
                # ignore href starts with #
                continue
            yield scrapy.Request(url, callback=self.parse)