小批量大小的Keras内存不足

Question

我只使用tensorflow库构建了一个自动编码器，网络形状为：

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_1 (InputLayer)         (None, 168, 120, 3)       0
_________________________________________________________________
flatten_1 (Flatten)          (None, 60480)             0
_________________________________________________________________
dense_1 (Dense)              (None, 1024)              61932544
_________________________________________________________________
dense_2 (Dense)              (None, 256)               262400
_________________________________________________________________
dense_3 (Dense)              (None, 1024)              263168
_________________________________________________________________
dense_4 (Dense)              (None, 60480)             61992000
_________________________________________________________________
reshape_1 (Reshape)          (None, 168, 120, 3)       0
=================================================================
Total params: 124,450,112
Trainable params: 124,450,112
Non-trainable params: 0
_________________________________________________________________

在这个仅仅使用tensorflow的项目中，我能够使用我的gpu进行无问题的训练，批量大小为128。我想用keras重新创建自动编码器，结果遇到内存不足异常，即使批处理大小为1。通过对这个问题的研究，我发现解决这个问题的最好办法是减少批量，但我不能再缩小它了。我的机器有2个GTX970卡在SLI中运行（CUDA不关心SLI），总共有8GB的内存。为什么我不能用keras来训练这个网络，即使我可以用tensorflow训练同一个批大小为64倍的网络？在

以下是相关代码：

常量：

^{pr2}$

初始化和保存：

# this is our input placeholder
input_img = Input(shape=(constants.HEIGHT,constants.WIDTH,constants.CHANNELS))
# flatten image into one dimension
flatten = Flatten()(input_img)
# hidden layer 1
hidden = Dense(constants.HIDDEN_WIDTH, activation='relu')(flatten)
# "encoded" is the encoded representation of the input
encoded = Dense(constants.ENCODING_WIDTH, activation='relu')(hidden)
# hidden layer 3
hidden = Dense(constants.HIDDEN_WIDTH, activation='relu')(encoded)
# "decoded" is the lossy reconstruction of the input
decoded = Dense(constants.NUM_INPUTS, activation='relu')(hidden)
# reshape to image dimensions
reshape = Reshape((constants.HEIGHT,constants.WIDTH,constants.CHANNELS))(decoded)

# this model maps an input to its reconstruction
autoencoder = Model(input_img, reshape)

autoencoder.summary()

autoencoder.compile(optimizer='adam', loss='mean_squared_error')

train_datagen = ImageDataGenerator(data_format='channels_last',
                                   rescale=1./255)

test_datagen = ImageDataGenerator(data_format='channels_last',
                                  rescale=1./255)

train_generator = train_datagen.flow_from_directory(
        constants.INPUT_PATH, 
        target_size=(constants.HEIGHT,constants.WIDTH),
        color_mode='rgb',
        class_mode='input',
        batch_size=constants.BATCH_SIZE)

validation_generator = test_datagen.flow_from_directory(
        constants.VALIDATION_PATH, 
        target_size=(constants.HEIGHT,constants.WIDTH),
        color_mode='rgb',
        class_mode='input',
        batch_size=constants.VALIDATION_SIZE)


autoencoder.fit_generator(train_generator,
        steps_per_epoch=constants.NUM_SAMPLES*1.0/constants.BATCH_SIZE,
        epochs=1,
        verbose=2,
        validation_data=validation_generator,
        validation_steps=constants.VALIDATION_SAMPLES*1.0/constants.VALIDATION_SIZE)


# Creates a HDF5 file 'my_model.h5'
autoencoder.save(constants.MODEL_PATH+constants.MODEL_FILE)
with open(constants.MODEL_PATH+constants.EPOCH_FILE, 'w') as f:
    f.write(str(1))

print("Done, model created in: " + constants.MODEL_PATH)

错误日志的一部分：

2019-01-29 16:40:10.522222: W tensorflow/core/common_runtime/bfc_allocator.cc:271] ***********************************************************************************************_____
2019-01-29 16:40:10.525191: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at matmul_op.cc:478 : Resource exhausted: OOM when allocating tensor with shape[60480,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
  File "init.py", line 53, in <module>
    validation_steps=constants.VALIDATION_SAMPLES*1.0/constants.VALIDATION_SIZE)
  File "C:\Users\dekke\Anaconda3\envs\tensorflow\lib\site-packages\keras\legacy\interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "C:\Users\dekke\Anaconda3\envs\tensorflow\lib\site-packages\keras\engine\training.py", line 1418, in fit_generator
    initial_epoch=initial_epoch)
  File "C:\Users\dekke\Anaconda3\envs\tensorflow\lib\site-packages\keras\engine\training_generator.py", line 217, in fit_generator
    class_weight=class_weight)
  File "C:\Users\dekke\Anaconda3\envs\tensorflow\lib\site-packages\keras\engine\training.py", line 1217, in train_on_batch
    outputs = self.train_function(ins)
  File "C:\Users\dekke\Anaconda3\envs\tensorflow\lib\site-packages\keras\backend\tensorflow_backend.py", line 2715, in __call__
    return self._call(inputs)
  File "C:\Users\dekke\Anaconda3\envs\tensorflow\lib\site-packages\keras\backend\tensorflow_backend.py", line 2675, in _call
    fetched = self._callable_fn(*array_vals)
  File "C:\Users\dekke\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1439, in __call__
    run_metadata_ptr)
  File "C:\Users\dekke\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 528, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1024,60480] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[{{node training/Adam/gradients/dense_4/MatMul_grad/MatMul_1}} = MatMul[T=DT_FLOAT, _class=["loc:@training/Adam/gradients/dense_4/MatMul_grad/MatMul"], transpose_a=true, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](dense_3/Relu, training/Adam/gradients/dense_4/Relu_grad/ReluGrad)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Answer 1

我经常得到这个使用水蟒tfÎgpu包与keras。我认为您要么是通过python脚本耗尽您的空闲内存，要么是tensorflow gpu试图一次分配大量内存：

我通常把这个放在我的输入项下，它很有效：

import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
sess = tf.Session(config = config)

# Check available GPU devices.
print("The following GPU devices are available: %s" % tf.test.gpu_device_name())

希望这有帮助。在