如何加快大规模出口？

Question

我正在使用python 3{}API导出一些海量日志。 我的代码基本上遵循splunkAPI准则：

import splunklib.client as client
import splunklib.results as results
import pandas as pd

kwargs_export = {"earliest_time": "2019-08-19T12:00:00.000-00:00",
                 "latest_time": "2019-08-19T14:00:00.000-00:00",
                 "search_mode": "normal"}

exportsearch_results = service.jobs.export(mysearchquery, **kwargs_export)

reader = results.ResultsReader(exportsearch_results)    

df = pd.DataFrame(list(reader))

但这是非常缓慢的

最后，我希望将搜索的输出作为csv存储到磁盘。有没有办法加快出口速度

谢谢

Answer 1

检查这个，因为它的工作

kwargs_export = {"earliest_time": "-1d",
                 "latest_time": "now",
                 "search_mode": "normal"}

service = client.connect(**args)

job = service.jobs.create(query, **kwargs_export)

with open(filename, 'wb') as out_f:

    try:

        job_results = job.results(output_mode="csv", count=0)

        for result in job_results:

             out_f.write(result)

    except :

        print("Session timed out. Reauthenticating")