我有一个脚本可以使用asyncio下载文件。我面临的问题是,当出现异常时,它会在任务中被捕获,但任务会继续并再次启动循环
在下载器中,实现了分页循环,内部循环尝试在3次尝试后停止,并且为了停止外部循环分页将在中引发并捕获一个异常,除非…此时协同程序应返回,但循环尝试会再次启动,但这次的值为2
只有当多个任务(coroutne下载器)同时运行时才会发生这种行为(当多个任务被提交到循环时),如果协程一个接一个地运行,则协程具有检测到的行为
async def downloader(session, url, spe_params, path, name, writer,
max_features=5000, sleeping=5):
try:
pagging = True
while pagging:
attemps = 3
while attemps:
# attemps reduced for looping
try:
async with session.get(url, params=p) as res:
if res.status != 200:
raise aiohttp.ClientError
res_json = await res.json()
except (aiohttp.ClientError, asyncio.TimeoutError,
json.JSONDecodeError) as e:
message = 'Error fetching {0}'.format(str(e))
attemps -= 1
await asyncio.sleep(sleeping)
else:
if res_json['totalFeatures'] == number_returned:
pagging = False
break
# Here the logic for pagination
if not attemps:
# attemps have been consumed
# Pagination stops and raise exception
raise DownloadError(message)
except DownloadError as e:
# Exception is catched. The coroutine should end but instead of that
# the loop "attemps" start again with the original value - 1
else:
# Json written to file
async def main():
loop_tasks = []
for id in field_id:
task = loop.create_task(downloader(session, ... )
done, pending = await asyncio.wait(loop_tasks,
return_when='ALL_COMPLETED')
目前没有回答
相关问题 更多 >
编程相关推荐