输送到后服务区的管道

Question

我需要一个管道来异步地将项目发送到存储服务。我想用类似FilePipeline的东西来做这个。FilePipeline有很多开销，因为它假设我想把文件保存到磁盘上，但是这里我只想把文件发布到一个存储API中。但是，它确实有一个生成请求的方法：get_media_requests()。在

我目前遇到FileException故障，我不知道如何消除保存到磁盘的组件。有没有办法让这一切顺利进行？在

class StoragePipeline(FilePipeline):


    access_token = os.environ['access_token']

    def get_media_requests(self, item, info):

        filename = item['filename']


        headers = {
            'Authorization': f'Bearer {self.access_token}',
            'Dropbox-API-Arg': f'{{"path": "/{filename}"}}',
            'Content-Type': 'application/octet-stream',
        }

        request = Request(
            method='POST',
            url='https://content.dropboxapi.com/2/files/upload',
            headers=headers,
            body=item['data'],

        )

        yield request


    def item_completed(self, results, item, info):

        return item

Answer 1

通过公开爬虫程序并直接安排请求，可以在管道中安排垃圾请求：

class MyPipeline(object):
    def __init__(self, crawler):
        self.crawler = crawler

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler)

    def process_item(self, item, spider):
        if item['some_extra_field']:  # check if we already did below
            return item
        req = scrapy.Request('some_url', self.check_deploy,
                             method='POST', meta={'item': item})
        self.crawler.engine.crawl(req, spider)
        return item

    def check_deploy(self, response):
        # if not 200 we might want to retry
        if response.status != 200: 
            return response.meta['item']