用于包装Altamira项目的群集日志记录的库。
cluster-logger的Python项目详细描述
python 3.6的集群日志库
这是Python3.6的一个库，它将帮助实现 集群服务和弹性搜索日志记录的要求。
要求
- Python3.6+
- pipenv
- 弹性搜索
- fluentd
- 基班纳
pipenv&pipfile管道文件
pipenv将包管理和virtualenv结合到一个工具中。
管道编号：https://docs.pipenv.org/basics
要安装：
$ pip install pipenv
pipfile将替换requirements.txt，它将在一个文件中同时指定依赖项和开发依赖项。
开发安装
# install dependencies pipenv install --dev # activate virtual environment pipenv shell # alternativly you can start your script in a virtual environment context pipenv run python main.py message that you want to send
ULM配置
默认配置使用环境变量配置记录器。
这些值是从ulm日志公共java实现的版本2.0中获取的。并非每个值都已实现，更改是表中最新的。
Repo
| Variable | Default Value | Description | Implemented |
|---|---|---|---|
| ULM_FLUENTD_HOST | localhost | host of the FluentD service | ✔️ |
| ULM_FLUENTD_PORT | 24224 | port of FluentD tcp "in_forward" | ✔️ |
| ULM_FLUENTD_LABEL_ENVS | comma separated list(with or without space) of other ENV vars to add as fields to FluentD logs | ✔️ | |
| ULM_FLUENTD_TAG | (project).(app) | tag to use when sending logs to fluentd, e.g. the MLA_IDENT string such as "Myapp.logs" | ✔️ |
| ULM_FLUENTD_BUFFER | 1048576 | an integer of the number of bytes to allow in the buffer. Excess will be dropped (default is 1MiB) | ❌ |
| ULM_FLUENTD_TIMEOUT | 3.0 | an float of the number of ms to allow for communication with FluentD. | ✔️ |
| ULM_FLUENTD_CLIENT | async | Whether to send async messages or not. This will send async as long as the value has async somewhere in the string. | ✔️ |
| ULM_FLUENTD_ENABLE_TIMEMS_FIELD | False | Enables an additional field to contain the millisecond timestamp | ❌ |
| ULM_FLUENTD_TIMEMS_FIELD | timems | the name of the field to contain the millisecond timestamp | ❌ |
独立使用（开发）
可能需要运行以下命令才能运行弹性搜索。
sudo sysctl -w vm.max_map_count=262144
局部
$ pipenv run python main.py -h usage: main.py [-h][-n HOST][-p PORT][-e [ENV [ENV ...]]][-o PROJECT][-a APP][-c COUNT] Console Application to test Cluster Logging using fluentd. positional arguments: message Message to be logged. optional arguments: -h, --help show this help message and exit -n HOST, --host HOST Hostname to listen on. (default=localhost) -p PORT, --port PORT Port number to bind to. (default=24224) -e [ENV [ENV ...]], --env [ENV [ENV ...]] Environment Keys to add to message. -o PROJECT, --project PROJECT Tag for the project -a APP, --app APP Tag for the application -c COUNT, --count COUNT The number of times to send message.
Docker
$ docker build -t TAG_FOR_THE_BUILD:VERSION . # all env values can be set with -e $ docker run -e host=fluentd \ -e port=24224 -e msg="message you want to send" TAG_FOR_THE_BUILD:VERSION # If you want to run the whole environment you can use docker-compose up $ docker-compose up
api
遵循python logger实现的日志级别。python.org
| Level | Numeric value |
|---|---|
| CRITICAL | 50 |
| ERROR | 40 |
| WARNING | 30 |
| INFO | 20 |
| DEBUG | 10 |
| NOTSET | 0 |
可以使用以下设置日志记录级别：
importlogginglogging.basicConfig(level=logging.WARNING)
设置日志级别将筛选低于设置日志级别的级别。对于日志级别，错误和堆栈之上的跟踪将添加到日志消息中。
标准配置
这些环境变量可以在系统范围内设置，也可以添加到.env文件中：
默认情况下，将提取3个环境变量：
HOST-运行容器的主机。APP_VERSION-应用程序的版本。LOGGER_VERBOSE-fluent日志库是否冗长。MARATHON_APP_ID-由DCO分配的ID。MARATHON_APP_DOCKER_IMAGE-容器是从中生成的图像。
有三种方法可以将应用程序属性获取到日志消息中。
- 传递键值对字典
- 传递容器环境中的键列表
- 或者传递带有键值对的json文件。
注意：
- 字典将覆盖环境和json文件中的值。
- 环境将覆盖json文件中的值。
使用量
启动您的VenV
# Linux / MacOSsource{NAME_OF_VENV}/bin/activate
REM Windows
{NAME_OF_VENV}\Scripts\activate
将群集记录器安装到您的VENV中。群集记录器已安装到PYPI中。
$ pip install cluster_logger
使用
importcluster_loggerimportlogging# Then init the ClusterLogger classenv_keys=['MARATHON_APP_LABELS','MARATHON_APP_RESOURCE_CPUS']props={'HOST','192.168.0.12'}logging.basicConfig(level=logging.DEBUG)# set logging levelcluster_logger.initLogger('Rasters',# Project name'intent_client',# Application name'fluentd',# Fluentd Host name (optional) will default to fluentd if nothing is passed in and HOST isn't set on the environment24224,# Fluentd Port Number(optional)'/path/to/settings.json',# JSON settings file (optional)env_keys,# Environment Keys (optional)props)# Specific Properties (optional)# in Module where you will logimportcluster_logger# General Loggerlogger=cluster_logger.getLogger(__name__)# Metric Loggermetrics=cluster_logger.getLogger(__name__+'.metrics',is_metric=True)logger.exception('Custom Message to send typically an error')logger.log(logging.INFO,{'Duration',58})
贡献
# bump the version that is stored in setup.py and cluster_logger/__init__.py# assuming pipenv install --dev was ran first $ pipenv run bumpversion minor # possible: major / minor / patch $ git push
欢迎加入QQ群-->： 979659372
