Stylegan2突然减速

Question

我正在使用stylegan2学习256x256大小的图像。 使用的命令如下。 python .\run_training.py --num-gpus=1 --data-dir="folder" --dataset=datasets256 --config=config-e --total-kimg=10000

我在KIMG322停下来，使用下面链接中的代码继续学习

https://github.com/ashirviskas/stylegan2/pull/1/commits/cc55d3796ea83f8a7d6fcf7f14166724bccd7c7e

python .\run_training.py --num-gpus=1 --data-dir="folder" --dataset=datasets256 --config=config-e --resume-pkl=network-snapshot-000322.pkl --resume-kimg=322.7

我在Anaconda中使用了Python 3.6

tensorflow（-gpu，-估计器，tensorboard）1.14.0

nvcc 10.0（+cudafe++.exe来自10.1）https://github.com/tensorflow/tensorflow/issues/27576#issuecomment-527900841

Ryzen 2700x|| gtx 1070 x 1|| 在win10上

network-snapshot-000322        time 18m 59s      fid50k 299.8156
tick 1     kimg 330.9    lod 0.00  minibatch 32   time 43m 22s      sec/tick 1368.6  sec/kimg 169.71  maintenance 1181.7 gpumem 4.9
tick 2     kimg 339.0    lod 0.00  minibatch 32   time 1h 06m 13s   sec/tick 1370.8  sec/kimg 169.99  maintenance 0.0    gpumem 4.9
tick 3     kimg 347.0    lod 0.00  minibatch 32   time 1h 29m 04s   sec/tick 1371.4  sec/kimg 170.07  maintenance 0.0    gpumem 4.9
tick 4     kimg 355.1    lod 0.00  minibatch 32   time 1h 51m 56s   sec/tick 1372.1  sec/kimg 170.15  maintenance 0.0    gpumem 4.9
tick 5     kimg 363.1    lod 0.00  minibatch 32   time 2h 14m 48s   sec/tick 1371.9  sec/kimg 170.13  maintenance 0.0    gpumem 4.9
tick 6     kimg 371.2    lod 0.00  minibatch 32   time 2h 37m 43s   sec/tick 1374.4  sec/kimg 170.44  maintenance 0.0    gpumem 4.9
tick 7     kimg 379.3    lod 0.00  minibatch 32   time 3h 00m 36s   sec/tick 1373.3  sec/kimg 170.30  maintenance 0.0    gpumem 4.9
tick 8     kimg 387.3    lod 0.00  minibatch 32   time 3h 23m 27s   sec/tick 1371.4  sec/kimg 170.06  maintenance 0.0    gpumem 4.9
tick 9     kimg 395.4    lod 0.00  minibatch 32   time 3h 46m 17s   sec/tick 1369.6  sec/kimg 169.84  maintenance 0.0    gpumem 4.9
tick 10    kimg 403.5    lod 0.00  minibatch 32   time 4h 09m 08s   sec/tick 1371.3  sec/kimg 170.05  maintenance 0.0    gpumem 4.9
network-snapshot-000403        time 18m 54s      fid50k 286.7292
tick 11    kimg 411.5    lod 0.00  minibatch 32   time 4h 51m 20s   sec/tick 1371.3  sec/kimg 170.05  maintenance 1160.0 gpumem 4.9
tick 12    kimg 419.6    lod 0.00  minibatch 32   time 5h 14m 14s   sec/tick 1374.9  sec/kimg 170.50  maintenance 0.0    gpumem 4.9
tick 13    kimg 427.7    lod 0.00  minibatch 32   time 5h 37m 06s   sec/tick 1371.6  sec/kimg 170.09  maintenance 0.0    gpumem 4.9
tick 14    kimg 435.7    lod 0.00  minibatch 32   time 6h 00m 00s   sec/tick 1373.6  sec/kimg 170.33  maintenance 0.0    gpumem 4.9
tick 15    kimg 443.8    lod 0.00  minibatch 32   time 6h 22m 54s   sec/tick 1374.2  sec/kimg 170.41  maintenance 0.0    gpumem 4.9
tick 16    kimg 451.9    lod 0.00  minibatch 32   time 6h 55m 23s   sec/tick 1948.9  sec/kimg 241.68  maintenance 0.0    gpumem 4.9
tick 17    kimg 459.9    lod 0.00  minibatch 32   time 7h 25m 43s   sec/tick 1820.1  sec/kimg 225.71  maintenance 0.2    gpumem 4.9
tick 18    kimg 468.0    lod 0.00  minibatch 32   time 7h 49m 25s   sec/tick 1421.4  sec/kimg 176.27  maintenance 0.0    gpumem 4.9
tick 19    kimg 476.0    lod 0.00  minibatch 32   time 8h 13m 11s   sec/tick 1426.2  sec/kimg 176.86  maintenance 0.0    gpumem 4.9
tick 20    kimg 484.1    lod 0.00  minibatch 32   time 8h 36m 41s   sec/tick 1410.7  sec/kimg 174.93  maintenance 0.0    gpumem 4.9
network-snapshot-000484        time 19m 44s      fid50k 275.0736
tick 21    kimg 492.2    lod 0.00  minibatch 32   time 9h 19m 55s   sec/tick 1381.2  sec/kimg 171.27  maintenance 1212.2 gpumem 4.9
tick 22    kimg 500.2    lod 0.00  minibatch 32   time 9h 43m 38s   sec/tick 1423.1  sec/kimg 176.48  maintenance 0.0    gpumem 4.9
tick 23    kimg 508.3    lod 0.00  minibatch 32   time 10h 07m 17s  sec/tick 1419.4  sec/kimg 176.01  maintenance 0.0    gpumem 4.9
tick 24    kimg 516.4    lod 0.00  minibatch 32   time 10h 30m 56s  sec/tick 1418.9  sec/kimg 175.96  maintenance 0.0    gpumem 4.9
tick 25    kimg 524.4    lod 0.00  minibatch 32   time 10h 56m 10s  sec/tick 1513.6  sec/kimg 187.70  maintenance 0.0    gpumem 4.9
tick 26    kimg 532.5    lod 0.00  minibatch 32   time 11h 20m 31s  sec/tick 1460.3  sec/kimg 181.09  maintenance 0.7    gpumem 4.9
tick 27    kimg 540.6    lod 0.00  minibatch 32   time 11h 45m 50s  sec/tick 1519.1  sec/kimg 188.38  maintenance 0.0    gpumem 4.9
tick 28    kimg 548.6    lod 0.00  minibatch 32   time 12h 17m 01s  sec/tick 1871.1  sec/kimg 232.04  maintenance 0.0    gpumem 4.9
tick 29    kimg 556.7    lod 0.00  minibatch 32   time 12h 46m 34s  sec/tick 1772.4  sec/kimg 219.79  maintenance 0.0    gpumem 4.9
tick 30    kimg 564.7    lod 0.00  minibatch 32   time 13h 14m 58s  sec/tick 1704.7  sec/kimg 211.40  maintenance 0.0    gpumem 4.9
network-snapshot-000564        time 20m 19s      fid50k 261.2363
tick 31    kimg 572.8    lod 0.00  minibatch 32   time 13h 59m 56s  sec/tick 1440.9  sec/kimg 178.68  maintenance 1256.9 gpumem 4.9
tick 32    kimg 580.9    lod 0.00  minibatch 32   time 14h 26m 20s  sec/tick 1583.7  sec/kimg 196.39  maintenance 0.2    gpumem 4.9
tick 33    kimg 588.9    lod 0.00  minibatch 32   time 15h 00m 07s  sec/tick 2026.0  sec/kimg 251.24  maintenance 0.9    gpumem 4.9
tick 34    kimg 597.0    lod 0.00  minibatch 32   time 18h 22m 56s  sec/tick 12167.1 sec/kimg 1508.82 maintenance 1.8    gpumem 4.9
tick 35    kimg 605.1    lod 0.00  minibatch 32   time 22h 25m 23s  sec/tick 14546.9 sec/kimg 1803.93 maintenance 0.0    gpumem 4.9
tick 36    kimg 613.1    lod 0.00  minibatch 32   time 1d 02h 07m   sec/tick 13306.1 sec/kimg 1650.06 maintenance 0.1    gpumem 4.9
tick 37    kimg 621.2    lod 0.00  minibatch 32   time 1d 05h 44m   sec/tick 13049.9 sec/kimg 1618.30 maintenance 0.0    gpumem 4.9