跟踪日志,提取数据,并将其发送给fluentd
flosculus的Python项目详细描述
Flosculus是一个work-in-progress脚本,用于跟踪旋转的日志文件,解析每一行,并将其发送到Fluentd。 它旨在替换fluentdin_tail功能的子集。
速成课程
首先,将flosculus安装到python路径中。
$ git clone git://github.com/iromli/flosculus.git
$ cd flosculus
$ python setup.py install
之后,您需要创建一个配置文件来运行flosculusd事件循环。 做这种无聊事情的方便方法是调用flosculusd --init > flosculus.ini命令。 这里有一个flosculus.ini的例子(您可以随意调用):
[flosculus]; the IP address (or host name) of the remote serverremote_host=127.0.0.1; the TCP port of the remote serverremote_port=24224; Each section with `path:/path/to/log` is a valid config[log:/var/log/nginx/access.log]; the labeltag=example.api.access; format to use, either use 'nginx' or custom regexformat=nginx
默认情况下,format选项与默认访问日志格式nginx匹配。 只要使用有效的python正则表达式,就可以更改格式。
[log:/var/log/nginx/access.log]format=(?P<remote>[^ ]*) (?P<host>[^ ]*) (?P<user>[^ ]*) \[(?P<time>[^\]]*)\] "(?P<method>\S+)(?: +(?P<path>[^\"]*) +\S*)?" (?P<code>[^ ]*) (?P<size>[^ ]*)(?: "(?P<referer>[^\"]*)" "(?P<agent>[^\"]*)")(?: (?P<request_time>[^ ]*) (?P<upstream_time>[^ ]*) (?P<pipe>[\.|p]))?
我有没有说过你可以有多个日志文件?只需复制整个log部分。
; Each section with `path:/path/to/log` is a valid config[log:/var/log/nginx/access.log]; the labeltag=example.api.access; format to use, either use 'nginx' or custom regexformat=nginx; Each section with `path:/path/to/log` is a valid config[log:/var/log/nginx/timed-combined.access.log]; the labeltag=test.api.access; format to use, either use 'nginx' or custom regexformat=(?P<remote>[^ ]*) (?P<host>[^ ]*) (?P<user>[^ ]*) \[(?P<time>[^\]]*)\] "(?P<method>\S+)(?: +(?P<path>[^\"]*) +\S*)?" (?P<code>[^ ]*) (?P<size>[^ ]*)(?: "(?P<referer>[^\"]*)" "(?P<agent>[^\"]*)")(?: (?P<request_time>[^ ]*) (?P<upstream_time>[^ ]*) (?P<pipe>[\.|p]))?; the IP address (or host name) of the remote server; use another Fluentd remote hostremote_host=10.0.0.1; the TCP port of the remote server; use another Fluentd remote portremote_port=24225
假设正确地编写了配置,运行事件循环以查看它的魔力(嗯,不是真的):
$ flosculusd -c flosculus.ini