Python Scrapy 301重定向

2024-09-29 08:15:17 发布

您现在位置:Python中文网/ 问答频道 /正文

我有一个小问题,在打印重定向的网址(新的网址后301重定向)时,刮一个给定的网站。我的想法是只印而不刮。我现在的代码是:

import scrapy
import os
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

class MySpider(CrawlSpider):
    name = 'rust'
    allowed_domains = ['example.com']
    start_urls = ['http://example.com']

    rules = (
        # Extract links matching 'category.php' (but not matching 'subsection.php')
        # and follow links from them (since no callback means follow=True by default).
        # Extract links matching 'item.php' and parse them with the spider's method parse_item
        Rule(LinkExtractor(), callback='parse_item', follow=True),
    )

    def parse_item(self, response):
        #if response.status == 301:
        print response.url

但是,这不会打印重定向的url。任何帮助都将不胜感激。

谢谢你。


Tags: fromimportparseresponselinksitemrule重定向