使用Python编写HDFS

Question

我试图用Python编写HDFS。 现在，我正在使用https://hdfscli.readthedocs.io/en/latest/quickstart.html 但对于大文件，我会说：

    File "/home/edge7/venv-dev/local/lib/python2.7/site-packages/hdfs/client.py", line 400, in write
    consumer(data)
  File "/home/edge7/venv-dev/local/lib/python2.7/site-packages/hdfs/client.py", line 394, in consumer
    auth=False,
  File "/home/edge7/venv-dev/local/lib/python2.7/site-packages/hdfs/client.py", line 179, in _request
    **kwargs
  File "/home/edge7/venv-dev/local/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/edge7/venv-dev/local/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/home/edge7/venv-dev/local/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', gaierror(-2, 'Name or service not known'))

我的写作代码非常简单：

^{pr2}$

有人能推荐一个合适的包来写HDFS吗？ 干杯

Answer 1

对于stacktrace，它似乎与安全相关。您确定需要使用不安全的客户机而不是Kerberos客户机吗？。另外，请记住，库只是HttpFs的一个绑定，因此使用Postman或CURL进行手动测试将允许您调试任何问题集群端。在