Tensorflow ObjectDetection=>验证tfrecord

Question

我用微软的VoTT标记了大约1000张图片，并将标记的图片导出到tfrecords。你知道吗

现在有了更快的\u rcnn \u inception \u v2 \u coco \u 2018 \u 01 \u 28，我每次在训练步骤50和3000之间都会得到一个NaN错误：

    I0824 21:33:31.837159 18516 coordinator.py:224] Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InvalidArgumentError'>, 2 root error(s) found.
  (0) Invalid argument: LossTensor is inf or nan. : Tensor had NaN values
         [[node CheckNumerics (defined at C:\tensorflow1\models\research\object_detection\legacy\trainer.py:322) ]]
  (1) Invalid argument: LossTensor is inf or nan. : Tensor had NaN values
         [[node CheckNumerics (defined at C:\tensorflow1\models\research\object_detection\legacy\trainer.py:322) ]]
         [[gradients/SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_3/Conv2d_0b_1x1/BatchNorm/FusedBatchNorm_grad/FusedBatchNormGrad/_2192]]
0 successful operations.
0 derived errors ignored.

Errors may have originated from an input operation.
Input Source operations connected to node CheckNumerics:
 total_loss (defined at C:\tensorflow1\models\research\slim\deployment\model_deploy.py:306)

Input Source operations connected to node CheckNumerics:
 total_loss (defined at C:\tensorflow1\models\research\slim\deployment\model_deploy.py:306)

Original stack trace for 'CheckNumerics':
  File "legacy/train.py", line 184, in <module>
    tf.app.run()
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\absl\app.py", line 300, in run
    _run_main(main, args)
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\absl\app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\tensorflow\python\util\deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "legacy/train.py", line 180, in main
    graph_hook_fn=graph_rewriter_fn)
  File "C:\tensorflow1\models\research\object_detection\legacy\trainer.py", line 322, in train
    total_loss = tf.check_numerics(total_loss, 'LossTensor is inf or nan.')
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\tensorflow\python\ops\gen_array_ops.py", line 1205, in check_numerics
    "CheckNumerics", tensor=tensor, message=message, name=name)
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\tensorflow\python\framework\ops.py", line 3616, in create_op
    op_def=op_def)
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\tensorflow\python\framework\ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()

Traceback (most recent call last):
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\tensorflow\python\client\session.py", line 1356, in _do_call
    return fn(*args)
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\tensorflow\python\client\session.py", line 1341, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\tensorflow\python\client\session.py", line 1429, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument: LossTensor is inf or nan. : Tensor had NaN values
         [[{{node CheckNumerics}}]]
  (1) Invalid argument: LossTensor is inf or nan. : Tensor had NaN values
         [[{{node CheckNumerics}}]]
         [[gradients/SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_3/Conv2d_0b_1x1/BatchNorm/FusedBatchNorm_grad/FusedBatchNormGrad/_2192]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "legacy/train.py", line 184, in <module>
    tf.app.run()
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\absl\app.py", line 300, in run
    _run_main(main, args)
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\absl\app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\tensorflow\python\util\deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "legacy/train.py", line 180, in main
    graph_hook_fn=graph_rewriter_fn)
  File "C:\tensorflow1\models\research\object_detection\legacy\trainer.py", line 416, in train
    saver=saver)
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\tensorflow\contrib\slim\python\slim\learning.py", line 775, in train
    train_step_kwargs)
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\tensorflow\contrib\slim\python\slim\learning.py", line 490, in train_step
    run_metadata=run_metadata)
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\tensorflow\python\client\session.py", line 950, in run
    run_metadata_ptr)
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\tensorflow\python\client\session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\tensorflow\python\client\session.py", line 1350, in _do_run
    run_metadata)
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\tensorflow\python\client\session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument: LossTensor is inf or nan. : Tensor had NaN values
         [[node CheckNumerics (defined at C:\tensorflow1\models\research\object_detection\legacy\trainer.py:322) ]]
  (1) Invalid argument: LossTensor is inf or nan. : Tensor had NaN values
         [[node CheckNumerics (defined at C:\tensorflow1\models\research\object_detection\legacy\trainer.py:322) ]]
         [[gradients/SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_3/Conv2d_0b_1x1/BatchNorm/FusedBatchNorm_grad/FusedBatchNormGrad/_2192]]
0 successful operations.
0 derived errors ignored.

Errors may have originated from an input operation.
Input Source operations connected to node CheckNumerics:
 total_loss (defined at C:\tensorflow1\models\research\slim\deployment\model_deploy.py:306)

Input Source operations connected to node CheckNumerics:
 total_loss (defined at C:\tensorflow1\models\research\slim\deployment\model_deploy.py:306)

Original stack trace for 'CheckNumerics':
  File "legacy/train.py", line 184, in <module>
    tf.app.run()
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\absl\app.py", line 300, in run
    _run_main(main, args)
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\absl\app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\tensorflow\python\util\deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "legacy/train.py", line 180, in main
    graph_hook_fn=graph_rewriter_fn)
  File "C:\tensorflow1\models\research\object_detection\legacy\trainer.py", line 322, in train
    total_loss = tf.check_numerics(total_loss, 'LossTensor is inf or nan.')
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\tensorflow\python\ops\gen_array_ops.py", line 1205, in check_numerics
    "CheckNumerics", tensor=tensor, message=message, name=name)
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\tensorflow\python\framework\ops.py", line 3616, in create_op
    op_def=op_def)
  File "C:\Users\hankypanky\AppData\Local\Continuum\anaconda3\envs\tensorflow2\lib\site-packages\tensorflow\python\framework\ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()

我读到，这可能是由于无效的标签图像注释。你知道吗

如果对现有的tfrecord文件进行了正确的注释，是否可以对其进行验证？ 或者是另一个问题？你知道吗