slackclient OSError:[Errno 24]打开的文件太多

Question

我在我的应用程序上添加slackclient以获得定期通知。 该应用程序在AWS Linux和docker容器中执行（从python:3.6.6-拉伸). 在

每当我试图在大约1000个循环中执行它时，进程失败了 在1000个循环之后，我得到了以下错误。在

我已经检查过stackoverflow上的其他案例，但它对我不起作用。希望我能从你的回答中得到一些启示。我个人想知道为什么会这样，解决核心问题，而不是绕道而行。在

错误

Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 321, in ssl_wrap_socket
OSError: [Errno 24] Too many open files

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connection.py", line 344, in connect
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 323, in ssl_wrap_socket
urllib3.exceptions.SSLError: [Errno 24] Too many open files

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='slack.com', port=443): Max retries exceeded with url: /api/chat.update (Caused by SSLError(OSError(24, 'Too many open files'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/scraping/notify.py", line 50, in update
  File "/home/scraping/notify.py", line 70, in _api_call
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/slackclient/client.py", line 184, in api_call
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/slackclient/server.py", line 349, in api_call
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/slackclient/slackrequest.py", line 84, in do
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/slackclient/slackrequest.py", line 116, in post_http_request
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/api.py", line 116, in post
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/api.py", line 60, in request
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/adapters.py", line 514, in send
requests.exceptions.SSLError: HTTPSConnectionPool(host='slack.com', port=443): Max retries exceeded with url: /api/chat.update (Caused by SSLError(OSError(24, 'Too many open files'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connection.py", line 159, in _new_conn
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/util/connection.py", line 57, in create_connection
  File "/usr/local/lib/python3.6/socket.py", line 745, in getaddrinfo
OSError: [Errno 24] Too many open files

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connection.py", line 301, in connect
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connection.py", line 168, in _new_conn
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7ff2d9158048>: Failed to establish a new connection: [Errno 24] Too many open files

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='slack.com', port=443): Max retries exceeded with url: /api/chat.postMessage (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff2d9158048>: Failed to establish a new connection: [Errno 24] Too many open files',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 33, in <module>
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/fire/core.py", line 127, in Fire
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/fire/core.py", line 366, in _Fire
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/fire/core.py", line 542, in _CallCallable
  File "main.py", line 29, in fasten
  File "/home/scraping/tasks/apps_info.py", line 199, in fasten
  File "/home/scraping/pipe.py", line 144, in _notify
  File "/home/scraping/notify.py", line 53, in update
  File "/home/scraping/notify.py", line 46, in resend
  File "/home/scraping/notify.py", line 70, in _api_call
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/slackclient/client.py", line 184, in api_call
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/slackclient/server.py", line 349, in api_call
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/slackclient/slackrequest.py", line 84, in do
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/slackclient/slackrequest.py", line 116, in post_http_request
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/api.py", line 116, in post
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/api.py", line 60, in request
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
  File "/root/.local/share/virtualenvs/home-oTyxYuqD/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='slack.com', port=443): Max retries exceeded with url: /api/chat.postMessage (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7ff2d9158048>: Failed to establish a new connection: [Errno 24] Too many open files',))

SlackClient部分

^{pr2}$

环境

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 15698
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65535
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 15698
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Answer 1

尝试使用 ulimit参数run docker容器

docker run -d  ulimit nofile=98304:98304 ...