在异步协同程序中捕获异常

2024-09-28 03:16:22 发布

您现在位置:Python中文网/ 问答频道 /正文

我有一个脚本可以使用asyncio下载文件。我面临的问题是,当出现异常时,它会在任务中被捕获,但任务会继续并再次启动循环

下载器中,实现了分页循环,内部循环尝试在3次尝试后停止,并且为了停止外部循环分页将在中引发并捕获一个异常,除非…此时协同程序应返回,但循环尝试会再次启动,但这次的值为2

只有当多个任务(coroutne下载器)同时运行时才会发生这种行为(当多个任务被提交到循环时),如果协程一个接一个地运行,则协程具有检测到的行为

async def downloader(session, url, spe_params, path, name, writer,
                     max_features=5000, sleeping=5):
    try:
        pagging = True
        while pagging:
            attemps = 3
            while attemps:
                # attemps reduced for looping
                try:
                    async with session.get(url, params=p) as res:
                        if res.status != 200:
                            raise aiohttp.ClientError
                        res_json = await res.json()

                except (aiohttp.ClientError, asyncio.TimeoutError,
                        json.JSONDecodeError) as e:
                    message = 'Error fetching {0}'.format(str(e))
                    attemps -= 1
                    await asyncio.sleep(sleeping)
                else:
                    if res_json['totalFeatures'] == number_returned:
                        pagging = False
                        break
                    # Here the logic for pagination

            if not attemps:
                # attemps have been consumed
                # Pagination stops and raise exception
                raise DownloadError(message)

    except DownloadError as e:
        # Exception is catched. The coroutine should end but instead of that
        # the loop "attemps" start again with the original value - 1

    else:
        # Json written to file

async def main():
    loop_tasks = []
    for id in field_id:
        task = loop.create_task(downloader(session, ... )

    done, pending = await asyncio.wait(loop_tasks,
                                               return_when='ALL_COMPLETED')


Tags: theloopasynciojsonforasyncifsession

热门问题