在Python中提取部分日志以导入到Ex

Question

我正在尝试使用regex获取日志（txt文件）的一部分，但我需要一些帮助。基本上日志是这样的：

Tue Feb 24 17:51:10.835 SRV02    NOTICE  Event Loop - noop
Tue Feb 24 17:51:10.835 SRV02    NOTICE     Exponential histogram:
Tue Feb 24 17:51:10.835 SRV02    NOTICE     hist[ 0]: <      0.001: 728941854
Tue Feb 24 17:51:10.835 SRV02    NOTICE  Event Loop - noop: samples: 728941854; avg: 0.00; min: 0.00; max: 0.00
Tue Feb 24 17:51:10.835 SRV02    NOTICE  Data Quality Monitor Thread Processing Time
Tue Feb 24 17:51:10.835 SRV02    NOTICE     Exponential histogram:
Tue Feb 24 17:51:10.835 SRV02    NOTICE     hist[ 4]: <      0.016:         3
Tue Feb 24 17:51:10.835 SRV02    NOTICE     hist[ 5]: <      0.032:        23
Tue Feb 24 17:51:10.835 SRV02    NOTICE     hist[ 6]: <      0.064:        14
Tue Feb 24 17:51:10.835 SRV02    NOTICE     hist[ 7]: <      0.128:         4
Tue Feb 24 17:51:10.835 SRV02    NOTICE     hist[ 8]: <      0.256:         6
Tue Feb 24 17:51:10.835 SRV02    NOTICE     hist[ 9]: <      0.512:         1
Tue Feb 24 17:51:10.835 SRV02    NOTICE     hist[10]: <      1.024:         2
Tue Feb 24 17:51:10.835 SRV02    NOTICE  Data Quality Monitor Thread Processing Time: samples: 53; avg: 0.08; min: 0.01; max: 0.67
Tue Feb 24 17:51:10.835 SRV02    NOTICE  Client Hugepage Memory:   649/4096 MB 
Tue Feb 24 17:51:10.836 SRV02    NOTICE  DQM: Num R: 0 RD: 0 ED: 0 W: 0 WH: 0 Q: 0 D: 0 DF: 0
Tue Feb 24 17:51:10.836 SRV02    NOTICE  Num G: 0 M: 0 S: 0 D: 0 U: 0 R: 0 N: 0
Tue Feb 24 17:51:10.836 SRV02    NOTICE  num_template_allocs                       =          4
Tue Feb 24 17:51:10.836 SRV02    NOTICE  num_template_frees                        =          0
Tue Feb 24 17:51:10.836 SRV02    NOTICE  num_internal_book_allocs                  =         24

我需要得到关于“指数直方图”的信息，所以，在这个例子中，我需要识别字符串“指数直方图”，并得到所有要导入到电子表格的“hist[…”。我还需要这些信息：

samples: XX; avg: X.XX; min: X.XX; max: X.XX

因此，在上面的示例中，我需要提取并重新排列这样的数据，其中“Event Loop-noop”和“data Quality Monitor Thread Processing Time”需要在每一行中重复，以便标识直方图：

Event Loop - noop;hist[ 0];0.001;728941854
Event Loop - noop;samples;728941854;avg;0.00;min;0.00;max;0.00
Data Quality Monitor Thread Processing Time;hist[ 4];0.016;3
Data Quality Monitor Thread Processing Time;hist[ 5];0.032;23
Data Quality Monitor Thread Processing Time;hist[ 6];0.064;14
(...)
Data Quality Monitor Thread Processing Time;hist[ 10];1.024;2
Data Quality Monitor Thread Processing Time;samples;53;avg;0.08;min;0.01;max;0.67

有人能帮我怎么做吗？谢谢您！你知道吗

Answer 1

在示例输出中，示例输入中不存在数据。具体来说，您的数据中有更多的"Data Quality Monitor Thread Processing Time"字符串。你想保留最近的缩进页眉吗？你知道吗

不管怎样，我认为使用几个不同的regex语句来提取数据会更容易，而不是试图制作一个包罗万象的语句：

import re
hists = re.findall(r'(hist\[\s\d+\]).*?(\d+\.\d+).*?(\d+)',input)
sample_avg_etc = re.findall(r'(samples): (\d+); (avg): (\d+\.\d+); (min): (\d+\.\d+); (max): (\d+\.\d+)',input)

如果需要保持本地标题与示例输出中显示的一样。我觉得你不想用正则表达式。相反，只需编写一个解析器来提取数据。你知道吗

您可以从剥离其Tue Feb 24 17:51:10.835 SRV02 NOTICE的每一行开始，然后逐行定位数据，跟踪最后一个头。请参阅注释，下面返回的正是您上面列出的内容：

import re

def parse(data):
    lines = data.split('\n')  # get the lines by splitting on the newline char
    lines = [line[len("Tue Feb 24 17:51:10.835 SRV02    NOTICE  "):]  for  line in lines]  # remove the number of characters equal to the logging info
    out = []
    header = ''
    for line in lines:
        if line.startswith('   '):
            if line.strip().startswith('hist'):
                out.append(header + ";" + extract_hist_data(line))  # outsource the specific extracting to a function for ease of readability
        else:                      # header/samples line
            if all(i in line for i in ("samples", "avg", "min", "max")):  # if the line contains all these keywords
                out.append(header + ";" + extract_stat_data(line))  # outsource the specific extracting to a function for ease of readability
            else:  # Treat as a header
                header = line
    return '\n'.join(out)

def extract_hist_data(line):
    data = re.findall(r'(hist\[\s*?\d+\]).*?(\d+\.\d+).*?(\d+)',line)
    if len(data) > 0:
        data = data[0]
    else:
        return ""
    return ';'.join(i for i in data)

def extract_stat_data(line):
    data = re.findall(r'(samples).*?(\d+).*?(avg).*?(\d+\.\d+).*?(min).*?(\d+\.\d+).*?(max).*?(\d+\.\d+)',line)
    if len(data) > 0:
        data = data[0]
    else:
        return ""
    return ';'.join(i for i in data)

def parse_log_file(log_file_path):
    with open(log_file_path,'r') as f:
        content = ''.join(i for i in f)
    return parse(content)

print parse_log_file('test.log')