如何使用自爬虫引擎.pause（）来自scrapy中的中间件

Question

我正在尝试从中间件暂停scrapy引擎（运行crawler）。

当我试着打电话的时候自爬虫引擎.unpause（）我收到此错误：

“cRetry”对象没有属性“crawler”

这是我的中间件。如何访问爬虫对象？

class cRetry(RetryMiddleware):

   errorCounter = 0


   def process_response(self, request, response, spider):
        if response.status in self.retry_http_codes:
            reason = response_status_message(response.status)
            return self._retry(request, reason, spider) or response
        elif "error" in response.body:
            self.errorCounter = self.errorCounter + 1
            if self.errorCounter >= 10:
                self.crawler.engine.pause()
                os.system("restart.sh")
                print "Reset"
                time.sleep(10)
                self.crawler.engine.unpause()
                self.errorCounter = 0
            reason ="Restart Required"
            return self._retry(request, reason, spider) or response
       ### end
        return response 

Answer 1

根据我的理解，您可以重写__init__和from_crawler方法，类似于：

class cRetry(RetryMiddleware):

    errorCounter = 0

    def __init__(self, crawler):
        super(cRetry, self).__init__(crawler.settings)
        self.crawler = crawler

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler)

    def process_response(self, request, response, spider):
        # ...

__init__的签名实际上似乎并不重要，entry point for the main library始终是{}。这是一个class method，它将类名作为第一个参数（然后使用它来调用构造函数）。在

Answer 2

谢谢aufziehvogel:—）

你的建议有点小意思。@需要添加classmethod，然后它就像一个魅力一样工作。在

class cRetry(RetryMiddleware):

    errorCounter = 0

    def __init__(self, crawler):
        super(cRetry, self).__init__(crawler.settings)
        self.crawler = crawler

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler)

    def process_response(self, request, response, spider):
        # ...