<p>我通常使用<code>tf.data.Dataset</code>和{<cd2>}和{<cd3>}。<code>Dataset.prefetch</code>意味着通常没有时间开销(只要CPU上的预处理比在GPU上运行网络花费的时间少)。如果你是跨多个GPU操作,你可能需要重新考虑,但下面的工作对我在单GPU系统上很好。在</p>
<p>为了简单起见,我假设您的所有图像都在磁盘上的不同文件中,尽管它可以很容易地适应zip存档或其他格式,如hdf5(不适用于<code>.tar</code>文件-不知道为什么,但我怀疑这是否是个好主意)</p>
<pre><code>import tensorflow as tf
from PIL import Image
def map_tf(path_tensor, label_tensor):
# path_tensor and label_tensor correspond to a single example
def map_np(path_str):
# path_str is just a normal string here
image = np.array(Image.load(path_str), dtype=np.uint8)
image = any_cv2_or_numpy_augmentations(image)
return image,
image, = tf.py_func(
map_np, (path_tensor,), Tout=(tf.uint8,), stateful=False)
# any tensorflow operations here.
image = tf.cast(image, tf.float32) / 255
image.set_shape((224, 224, 3))
return image, label
paths, labels = load_image_paths_and_labels()
dataset = tf.data.Dataset.from_tensor_slices((paths, labels))
if is_training:
shuffle_buffer = len(paths) # full shuffling - can be shorter
dataset = dataset.shuffle(shuffle_buffer).repeat()
dataset = dataset.map(map_tf_fn, num_parallel_calls=8)
dataset = dataset.batch(batch_size)
dataset = dataset.prefetch(1)
# play with the following if you want - not finalized API, and only in
# more recent version of tensorflow
# dataset = dataset.apply(tf.contrib.data.prefetch_to_device('/gpu:0'))
image_batch, label_batch = dataset.make_one_shot_iterator().get_next()
</code></pre>
<p>您也可以在tensorflow中进行解码,并直接在<code>py_func</code>中使用<code>any_cv2_or_numpy_augmentations</code>(尽管您没有避免您在问题中提到的张量->;numpy->;张量舞蹈)。我怀疑你会注意到性能的不同。在</p>
<p>有关更多选项,请查看<a href="https://stackoverflow.com/questions/45828616/streaming-large-training-and-test-files-into-tensorflows-dnnclassifier/45829855#45829855">this answer</a>。在</p>