使用Python日志模块将日志保存到S3,如何为所有模块捕获所有级别的日志?

2024-10-02 02:24:21 发布

您现在位置:Python中文网/ 问答频道 /正文

这是我第一次在Python(3.7)中使用日志模块。我的代码使用导入的模块,这些模块也有自己的日志语句。当我第一次向代码中添加日志语句时,我没有使用getLogger()。我只是使用了logging.basicConfig(filename)并直接调用logger.debug()来记录语句。当我这样做时,来自脚本和所有导入模块的所有日志一起输出到同一个文件中

现在我需要将代码转换为将日志保存到s3而不是文件。我尝试了How Can I Write Logs Directly to AWS S3 from Memory Without First Writing to stdout? (Python, boto3) - Stack Overflow中提到的解决方案,但有两个问题:

  1. 当我检查s3时,输出中没有“前缀”
  2. 只显示INFO语句。我的印象是logging.basicConfig(level=logging.INFO)会使它输出级别信息上的所有日志,但我只看到了信息。此外,只有信息日志会被打印到stdout,而在所有级别之前。我不知道为什么“前缀”不见了

from psaw import PushshiftAPI
api = PushshiftAPI()
import time
import logging
import boto3
import io
import atexit

def write_logs(body, bucket, key):
    s3 = boto3.client("s3")
    s3.put_object(Body=body.getvalue(), Bucket=bucket, Key=key)

logging.basicConfig(level=logging.INFO)
log = logging.getLogger()
log_stringio = io.StringIO()
handler = logging.StreamHandler(log_stringio)
log.addHandler(handler)

def collectRange(sub,start,end):
    atexit.register(write_logs, body=log_stringio, bucket="<...>", key=f'{sub}/log.txt')
    s3 = boto3.resource('s3')
    object = s3.Object('<...>', f'{sub}/{sub}@{start}-{end}.csv')
    now = time.time()
    logging.info(f'Start Time:{now}')
    logging.debug('First request')
    gen = api.search_comments(after=start, before=end,<...>, subreddit=sub)
    r=next(gen)
    <...>
    quit()

输出:

Found credentials in shared credentials file: ~/.aws/credentials
Start Time:1591310443.7060978
https://api.pushshift.io/reddit/comment/search?<...>
https://api.pushshift.io/reddit/comment/search?<...>

期望输出:

INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
INFO:root:Start Time:1591310443.7060978
DEBUG:root:First request
INFO:psaw.PushshiftAPI:https://api.pushshift.io/reddit/comment/search?<...>
DEBUG:psaw.PushshiftAPI:<whatever is usually here>
DEBUG:psaw.PushshiftAPI:<whatever is usually here>
INFO:psaw.PushshiftAPI:https://api.pushshift.io/reddit/comment/search?<...>
DEBUG:psaw.PushshiftAPI:<whatever is usually here>
DEBUG:psaw.PushshiftAPI:<whatever is usually here>

感谢您的帮助。谢谢


Tags: 模块iodebugimportinfologapisearch

热门问题