scrapy中未处理或不允许HTTP状态代码

Question

我是scrapy新手，遇到以下错误“HTTP状态代码未处理或不允许”，我已经尝试了this解决方案

myspider.py

import scrapy
class SubmitformSpider(scrapy.Spider):
    name = "formsubmit_xpath"
    website_url = "https://www.knowyourgst.com/gst-number-search/"
    start_urls = ["https://www.knowyourgst.com/gst-number-search/"]

    def parse(self,response):
        # extract csrf_token value
        token = response.css('input[name="csrfmiddlewaretoken"]::attr(value)').extract_first()
        # create a dictionary for form values 
        data = {
            'csrfmiddlewaretoken':token,
            'gstnum':'27ARFPS6606L1ZA'
        }     
        # submit a post request to it.
        headers= {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0'}
        yield scrapy.FormRequest(url=self.website_url, formdata = data, callback = self.parse_formrequest, headers=headers)

    def parse_formrequest(self,response):
        yield{
            'BusinessName' : quote.xpath('/html/body/div[1]/div[2]/div[2]/div/table/tbody/tr[1]/td[2]/text()').extract_first()
        }

设置.py

USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.92 Safari/537.36'

日志

2020-04-20 18:23:14 [scrapy.utils.log] INFO: Scrapy 1.8.0 started (bot: formsubmit_getresult)
2020-04-20 18:23:14 [scrapy.utils.log] INFO: Versions: lxml 4.3.4.0, libxml2 2.9.9, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 19.10.0, Python 3.7.3 (default, Mar 27 2019, 22:11:17) - [GCC 7.3.0], pyOpenSSL 19.0.0 (OpenSSL 1.1.1f  31 Mar 2020), cryptography 2.7, Platform Linux-5.3.0-46-generic-x86_64-with-debian-buster-sid
2020-04-20 18:23:14 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'formsubmit_getresult', 'NEWSPIDER_MODULE': 'formsubmit_getresult.spiders', 'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['formsubmit_getresult.spiders'], 'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.92 Safari/537.36'}
2020-04-20 18:23:14 [scrapy.extensions.telnet] INFO: Telnet Password: 0d70cf8b14bad003
2020-04-20 18:23:14 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.logstats.LogStats']
2020-04-20 18:23:14 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2020-04-20 18:23:14 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2020-04-20 18:23:14 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2020-04-20 18:23:14 [scrapy.core.engine] INFO: Spider opened
2020-04-20 18:23:14 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-04-20 18:23:14 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-04-20 18:23:14 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://www.knowyourgst.com/robots.txt/> from <GET https://www.knowyourgst.com/robots.txt>
2020-04-20 18:23:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.knowyourgst.com/robots.txt/> (referer: None)
2020-04-20 18:23:14 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.knowyourgst.com/gst-number-search/> (referer: None)
2020-04-20 18:23:15 [scrapy.core.engine] DEBUG: Crawled (403) <POST https://www.knowyourgst.com/gst-number-search/> (referer: https://www.knowyourgst.com/gst-number-search/)
2020-04-20 18:23:15 [scrapy.core.scraper] DEBUG: Scraped from <403 https://www.knowyourgst.com/gst-number-search/>
{'BusinessName': None}
2020-04-20 18:23:15 [scrapy.core.engine] INFO: Closing spider (finished)
2020-04-20 18:23:15 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 1586,
 'downloader/request_count': 4,
 'downloader/request_method_count/GET': 3,
 'downloader/request_method_count/POST': 1,
 'downloader/response_bytes': 6620,
 'downloader/response_count': 4,
 'downloader/response_status_count/200': 2,
 'downloader/response_status_count/301': 1,
 'downloader/response_status_count/403': 1,
 'elapsed_time_seconds': 1.018804,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2020, 4, 20, 12, 53, 15, 113726),
 'item_scraped_count': 1,
 'log_count/DEBUG': 5,
 'log_count/INFO': 10,
 'memusage/max': 54525952,
 'memusage/startup': 54525952,
 'request_depth_max': 1,
 'response_received_count': 3,
 'robotstxt/request_count': 1,
 'robotstxt/response_count': 1,
 'robotstxt/response_status_count/200': 1,
 'scheduler/dequeued': 2,
 'scheduler/dequeued/memory': 2,
 'scheduler/enqueued': 2,
 'scheduler/enqueued/memory': 2,
 'start_time': datetime.datetime(2020, 4, 20, 12, 53, 14, 94922)}
2020-04-20 18:23:15 [scrapy.core.engine] INFO: Spider closed (finished)

Answer 1

试试这个

token = response.css('input[name="csrfmiddlewaretoken"]::attr(value)').extract_first()