映射到tensorflow数据集，并对tf.train.Feature（字节字符串列表）进行变异

Question

我有一个功能是字节字符串列表，例如

data = [b"lksjdflksdjfdlk", b"owiueroiewuroi.skjdf", b"oweiureoiwlkapq"]

下面是创建、写出和读回并解析tfrecord的示例代码

>>> data = [b"lksjdflksdjfdlk", b"owiueroiewuroi.skjdf", b"oweiureoiwlkapq"]
>>> feature = tf.train.Feature(bytes_list=tf.train.BytesList(value=data))
>>> feature
{'raws': bytes_list {
   value: "lksjdflksdjfdlk"
   value: "owiueroiewuroi.skjdf"
   value: "oweiureoiwlkapq"
 }}
>>> example = tf.train.Example(features=features).SerializeToString()
>>> with tf.io.TFRecordWriter("/tmp/out.tfrecord") as writer:
        writer.write(example)
>>> # Now read it back in and parse thee example
>>> feature_desc = {'raws': tf.io.FixedLenFeature([], tf.string)}
>>> def _parse(example):
        return tf.io.parse_single_example(example, feature_desc)
>>> ds = tf.data.TFRecordDataset(["/tmp/out.tfrecord"])
>>> parsed = ds.map(_parse)
>>> @tf.function
    def upper(x):
        x['raws'] = [s.upper() for s in x['raws']]
>>> parsed.map(upper)

这会导致以下错误：

OperatorNotAllowedInGraphError: in user code:

    <ipython-input-33-be19a774366f>:3 upper  *
        x['raws'] = [s.upper() for s in x['raws']]
    /data/jkyle/venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py:503 __iter__
        self._disallow_iteration()
    /data/jkyle/venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py:496 _disallow_iteration
        self._disallow_when_autograph_enabled("iterating over `tf.Tensor`")
    /data/jkyle/venv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py:474 _disallow_when_autograph_enabled
        " indicate you are trying to use an unsupported feature.".format(task))

    OperatorNotAllowedInGraphError: iterating over `tf.Tensor` is not allowed: AutoGraph did convert this function. This might indicate you are trying to use an unsupported feature.

对于完整上下文，该列表是本机不支持的原始图像格式的字节字符串。每个原始图像都是一个帧。我需要迭代列表，转换成jpeg格式，然后将它们堆叠成一个三维数组。转换将需要由OpenCV完成。如此原始->；jpeg->；numpy矩阵，例如

输入：[b'raw1'，b'raw2'，b'raw3'] 输出：形状的图像阵列（19201080,3）

但是，当然，在我弄清楚如何迭代列表之前，我不能做任何这些

Answer 1

是的，正如错误所暗示的那样，在张量上迭代是不受支持的。这是一个相当笼统的问题，可能无法回答您的特定问题，但您可以使用： tf.unstack对其进行迭代

它将秩-R张量的给定维数分解为秩-（R-1）张量

因此，在每个张量上加1看起来像：

import tensorflow as tf
x = tf.placeholder(tf.float32, shape=(None, 10))
x_unpacked = tf.unstack(x) # defaults to axis 0, returns a list of tensors

processed = [] # this will be the list of processed tensors
for t in x_unpacked:
    # do whatever
    result_tensor = t + 1
    processed.append(result_tensor)

output = tf.concat(processed, 0)

with tf.Session() as sess:
    print(sess.run([output], feed_dict={x: np.zeros((5, 10))}))

显然，您可以进一步将列表中的每个张量解压为单个元素来处理它。不过，为了避免大量嵌套的解包，您可以尝试先用tf.reshape(x, [-1])展平x，然后像这样循环

flattened_unpacked = tf.unstack(tf.reshape(x, [-1])
for elem in flattened_unpacked:
    process(elem)

在这种情况下，元素是标量