使用gsutil cp（Google云存储）时内存不足/无法启动新线程

Question

我正在使用googlecloudsdk中的gsutil cp命令来检索存储在Google云存储桶中的一些文件。在

输出如下：

gsutil -m cp gs://my-bucket/spark-output/* .
Copying gs://my-bucket/spark-output/_SUCCESS...
Copying gs://my-bucket/spark-output/part-00000...
Copying gs://my-bucket/spark-output/part-00001...
Copying gs://my-bucket/spark-output/part-00002...
Copying gs://my-bucket/spark-output/part-00003...
can't start new threadGiB/ 13.5 GiB]  41% Done   7.1 MiB/s ETA 00:19:03
can't start new threadGiB/ 13.5 GiB]  41% Done   7.1 MiB/s ETA 00:19:03
can't start new threadGiB/ 13.5 GiB]  41% Done   7.1 MiB/s ETA 00:19:03
can't start new threadGiB/ 13.5 GiB]  41% Done   7.2 MiB/s ETA 00:18:54
can't start new threadGiB/ 13.5 GiB]  41% Done   7.2 MiB/s ETA 00:19:01
can't start new threadGiB/ 13.5 GiB]  41% Done   7.2 MiB/s ETA 00:19:01
can't start new threadGiB/ 13.5 GiB]  41% Done   7.6 MiB/s ETA 00:17:48
can't start new threadGiB/ 13.5 GiB]  41% Done   7.7 MiB/s ETA 00:17:41
can't start new thread
can't start new threadGiB/ 13.5 GiB]  41% Done   7.7 MiB/s ETA 00:17:30
can't start new threadGiB/ 13.5 GiB]  45% Done   6.9 MiB/s ETA 00:18:26
can't start new threadGiB/ 13.5 GiB]  45% Done   7.9 MiB/s ETA 00:15:52
can't start new threadGiB/ 13.5 GiB]  45% Done   7.9 MiB/s ETA 00:15:50
can't start new threadGiB/ 13.5 GiB]  45% Done   7.9 MiB/s ETA 00:15:50
\ [26/74 files][  6.2 GiB/ 13.5 GiB]  45% Done   7.1 MiB/s ETA 00:17:43
out of memory
out of memory
out of memory
out of memory
out of memory
out of memory
out of memory
out of memory
out of memory
out of memory
out of memory
out of memory
out of memory
out of memory
out of memorys][  6.2 GiB/ 13.5 GiB]  45% Done   5.4 MiB/s ETA 00:23:17
out of memory
out of memory
out of memory


out of memory
- [26/74 files][  6.2 GiB/ 13.5 GiB]  45% Done   4.8 MiB/s ETA 00:26:10
out of memory
out of memory

\
out of memorys][  6.2 GiB/ 13.5 GiB]  45% Done   1.5 MiB/s ETA 01:22:50
| [26/74 files][  6.2 GiB/ 13.5 GiB]  45% Done 942.6 KiB/s ETA 02:15:55

out of memory
out of memorys][  6.2 GiB/ 13.5 GiB]  45% Done 119.0 KiB/s ETA 17:56:57
CommandException: Some components of .\part-00038 were not downloaded successfully. Please retry this download.

- [26/74 files][  6.2 GiB/ 13.5 GiB]  45% Done 164.4 KiB/s ETA 12:59:04
out of memory
\ [26/74 files][  6.2 GiB/ 13.5 GiB]  45% Done 164.5 KiB/s ETA 12:58:31
out of memory
CommandException: Some components of .\part-00035 were not downloaded successfully. Please retry this download.
/
CommandException: Some components of .\part-00027 were not downloaded successfully. Please retry this download.
CommandException: Some components of .\part-00040 were not downloaded successfully. Please retry this download.
CommandException: Some components of .\part-00025 were not downloaded successfully. Please retry this download.
CommandException: Some components of .\part-00031 were not downloaded successfully. Please retry this download.
CommandException: Some components of .\part-00033 were not downloaded successfully. Please retry this download.
CommandException: Some components of .\part-00037 were not downloaded successfully. Please retry this download.
CommandException: Some components of .\part-00039 were not downloaded successfully. Please retry this download.
CommandException: Some components of .\part-00036 were not downloaded successfully. Please retry this download.
CommandException: Some components of .\part-00026 were not downloaded successfully. Please retry this download.
CommandException: Some components of .\part-00034 were not downloaded successfully. Please retry this download.
can't start new thread
CommandException: Some components of .\part-00028 were not downloaded successfully. Please retry this download.
CommandException: Some components of .\part-00029 were not downloaded successfully. Please retry this download.
can't start new threadGiB/ 13.5 GiB]  45% Done     0.0 B/s
can't start new thread
can't start new thread
can't start new thread
can't start new thread
can't start new thread
can't start new thread
can't start new thread
CommandException: 47 files/objects could not be transferred.

运行没有-m参数的命令仍然会崩溃。 我的googlecloudsdk是最新的，它运行在windows10上。 我的网络连接稳定。在

谢谢你的帮助。在