此工具允许您在日志数据平台上获取所有冷存储存档的本地副本。
ldp-archive-mirror的Python项目详细描述
此工具允许您在日志数据平台上获取所有冷存储存档的本地副本：
[2018-12-14 17:20:00,200 INFO LDP Mirror] Create local DB if not exists [2018-12-14 17:20:00,204 INFO LDP Mirror] Looking for new archives of stream e9397556-31d1-4d4d-b2bd-e5367b522cc8 [2018-12-14 17:20:05,190 INFO LDP Mirror] Service ldp-jb-52560 found for stream e9397556-31d1-4d4d-b2bd-e5367b522cc8 [2018-12-14 17:20:05,579 INFO LDP Mirror] Archive 9fb75957-2cde-435e-bdd9-6dfd33663f2c added to cache [2018-12-14 17:20:05,850 INFO LDP Mirror] Archive 11d8630a-7b38-4fa4-9d7c-dfd17b0b00f6 added to cache ... [2018-12-14 17:20:08,392 INFO LDP Mirror] Directory mirror/e9397556-31d1-4d4d-b2bd-e5367b522cc8 created [2018-12-14 17:20:08,535 INFO LDP Mirror] Archive e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-10.zst missing, download scheduled [2018-12-14 17:20:08,635 INFO LDP Mirror] Archive e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-11.zst missing, download scheduled ... [2018-12-14 17:20:09,535 INFO LDP Mirror] Will retry to download e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-10.zst after 596 seconds [2018-12-14 17:20:09,745 INFO LDP Mirror] Will retry to download e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-11.zst after 598 seconds ... [2018-12-14 17:20:10,927 INFO LDP Mirror] Archive e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-10.zst downloaded [2018-12-14 17:20:11,655 INFO LDP Mirror] Sha256 OK on e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-10.zst [2018-12-14 17:20:10,927 INFO LDP Mirror] Archive e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-11.zst downloaded [2018-12-14 17:20:11,655 INFO LDP Mirror] Sha256 OK on e9397556-31d1-4d4d-b2bd-e5367b522cc8/2018-12-11.zst
快速启动
首先，使用pip：
pip install -U ldp-archive-mirror
然后可以使用二进制ldp-mirror：
usage: ldp-mirror [-h] [--app-key KEY] [--app-secret SECRET]
[--consumer-key KEY] [--ovh-region REGION] [--db DIR]
[--mirror DIR] [--ldp-host HOST] [--ldp-token TOKEN]
STREAM_ID [STREAM_ID ...]
LDP archive Mirror CLI - 0.1.0
positional arguments:
STREAM_ID LDP Stream UUIDs
optional arguments:
-h, --help show this help message and exit
--app-key KEY OVH application key (default: None)
--app-secret SECRET OVH application secret (default: None)
--consumer-key KEY OVH consumer key (default: None)
--ovh-region REGION OVH region (default: ovh-eu)
--db DIR Where to place the local sqlite database (default: db)
--mirror DIR Where to place your archives (default: mirror)
--ldp-host HOST If set, push logs of the current application to given
LDP hostname
--ldp-token TOKEN If set, push logs of the current application to
associated LDP stream token
设置
一。创建应用程序
要与api交互，应用程序需要使用
application_key和一个application_secret。为了得到他们，你需要
注册你的申请。根据您计划使用的API，请访问：
- OVH Europe
- OVH US
- OVH North-America
- So you Start Europe
- So you Start North America
- Kimsufi Europe
- Kimsufi North America
在限制步骤中，我们邀请您设置以下访问规则：
curl -XPOST -H"X-Ovh-Application: OVH_API_AK" -H "Content-type: application/json" \
https://eu.api.ovh.com/1.0/auth/credential -d '{
"accessRules": [
{
"method": "GET",
"path": "/dbaas/logs"
},
{
"method": "GET",
"path": "/dbaas/logs/*/output/graylog/stream"
},
{
"method": "GET",
"path": "/dbaas/logs/*/output/graylog/stream/*/archive*"
},
{
"method": "POST",
"path": "/dbaas/logs/*/output/graylog/stream/*/archive/*/url"
}
],
"redirection":"https://www.mywebsite.com/"
}'
创建后，您将获得一个application key（ovh_api_ak）和一个application 秘密（ovh_api_as）。
2.环境变量
可以使用环境设置默认的cli值：
| Cli parameter | Name | About |
|---|---|---|
| –app-key | OVH_API_AK | OVH application key |
| –app-secret | OVH_API_AS | OVH application secret |
| –consumer-key | OVH_API_CK | OVH customer key |
| –ovh-region | OVH_API_REGION | OVH api location. Default: ovh-eu |
| –db | DB_DIRECTORY | Where to place the local sqlite database. Default: db |
| –mirror | MIRROR_DIRECTORY | Where to place your archives. Default: mirror |
| –ldp-host | LDP_HOST | If set, push logs of the current application to given LDP hostname |
| –ldp-token | LDP_TOKEN | If set, push logs of the current application to associated LDP stream token |
三。启动
设置完所有必需的环境变量后，启动如下过程：
$ ldp-mirror STREAM_ID [STREAM_ID ...]
这将：
- 查找与给定stream\u id关联的ldp服务（s）
- 使用API上找到的所有存档填充本地缓存
- 为每个用户请求一个临时下载URL
- 到达解封时间后下载文件
- 每小时询问API是否有新的存档可用
从源创建Docker图像
由于这个应用程序应该无限期地保持活动状态，从Docker守护进程启动它看起来很明显。
要从源构建图像，请使用给定的Makefile：
$ git clone https://github.com/ovh/ldp-archive-mirror $ cd ldp-archive-mirror $ make build-docker
运行它：
$ docker run -v /my_backup/mirror/:/data/mirror -v /my_backup/db:/data/db \ -e OVH_API_AK=MY_OVH_AK -e OVH_API_AS=MY_OVH_AS -e OVH_API_CK=MY_OVH_CK \ -t MY_LDP_STREAM_ID_1 MY_LDP_STREAM_ID_2
要求
- python>；=3.3
项目链接
许可证
欢迎加入QQ群-->： 979659372
