为什么Python停止使用GPU并在runtim中切换到CPU

Question

我一直在用这个：

os.environ["CUDA_VISIBLE_DEVICES"] = "1"

为了在GPU上运行。从今天起它一直正常工作。你知道吗

现在的问题是，在运行时的中间，我的程序停止使用GPU并切换到CPU，所以它变得太慢了。你知道吗

你知道为什么会这样吗？你知道吗

执行nvidia smi时的输出：

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.67       Driver Version: 418.67       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 970     On   | 00000000:01:00.0  On |                  N/A |
|  0%   42C    P8    14W / 200W |    363MiB /  4039MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K40c          On   | 00000000:05:00.0 Off |                    0 |
| 35%   74C    P0   136W / 235W |  11011MiB / 11441MiB |     94%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1037      G   /usr/lib/xorg/Xorg                            20MiB |
|    0      1150      G   /usr/bin/gnome-shell                          12MiB |
|    0      7430      G   /usr/lib/xorg/Xorg                           166MiB |
|    0      7560      G   /usr/bin/gnome-shell                         158MiB |
|    1     13772      C   python3                                    10998MiB |
+-----------------------------------------------------------------------------+

然后，当它开始运行太慢时：

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.67       Driver Version: 418.67       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 970     On   | 00000000:01:00.0  On |                  N/A |
|  0%   42C    P8    14W / 200W |    363MiB /  4039MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K40c          On   | 00000000:05:00.0 Off |                    0 |
| 35%   69C    P0    63W / 235W |  11011MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1037      G   /usr/lib/xorg/Xorg                            20MiB |
|    0      1150      G   /usr/bin/gnome-shell                          12MiB |
|    0      7430      G   /usr/lib/xorg/Xorg                           166MiB |
|    0      7560      G   /usr/bin/gnome-shell                         158MiB |
|    1     13772      C   python3                                    10998MiB |
+-----------------------------------------------------------------------------+