尝试训练时CNTK崩溃RCNN无法打开fi

Question

当我尝试使用CNTK FastRCNN示例来训练自定义数据集时，我得到一个错误，即它无法访问文件。文件可以手动打开。 这是奇怪的，因为它不是第一个文件之一，但接近尾声。我不认为这是缺乏资源，因为钻机是相当强大的，但我不能肯定。这是在跑步之后

$ python A2_RunWithPyModel.py


  (cntk) alex989898@Safe:~/boi/CNTK/Examples/Image/Detection/FastRCNN$ python A2_RunWithPyModel.py
    --------------------------------------------------------------
    2017-05-20 08:12:42
    PARAMETERS: datasetName = Grocery
    PARAMETERS: cntk_nrRois = 100
    -------------------------------------------------------------------
    Build info: 

            Built time: Apr 21 2017 07:23:46
            Last modified date: Tue Apr 18 12:39:07 2017
            Build type: release
            Build target: GPU
            With 1bit-SGD: no
            With ASGD: yes
            Math lib: mkl
            CUDA_PATH: /usr/local/cuda-8.0
            CUB_PATH: /usr/local/cub-1.4.1
            CUDNN_PATH: /usr/local/cudnn-5.1
            Build Branch: HEAD
            Build SHA1: f190457aecbfe07c3fe27c89f36108bc2488727b
            Built by Source/CNTK/buildinfo.h$$0 on 2ec53482eb0b
            Build Path: /home/philly/jenkins/workspace/CNTK-Build-Linux
            MPI distribution: Open MPI
            MPI version: 1.10.3
    -------------------------------------------------------------------
    Training Fast R-CNN model for 20 epochs.
    Training 54550532 parameters in 6 parameter tensors.
    Traceback (most recent call last):
      File "A2_RunWithPyModel.py", line 213, in <module>
        trained_model = train_fast_rcnn()
      File "A2_RunWithPyModel.py", line 168, in train_fast_rcnn
        data = minibatch_source.next_minibatch(min(mb_size, epoch_size-sample_count), input_map=input_map)
      File "/home/alex989898/.conda/envs/cntk/lib/python3.4/site-packages/cntk/internal/swig_helper.py", line 69, in wrapper
        result = f(*args, **kwds)
      File "/home/alex989898/.conda/envs/cntk/lib/python3.4/site-packages/cntk/io/__init__.py", line 340, in next_minibatch
        partition_index, device)
      File "/home/alex989898/.conda/envs/cntk/lib/python3.4/site-packages/cntk/cntk_py.py", line 2426, in get_next_minibatch
        return _cntk_py.MinibatchSource_get_next_minibatch(self, *args)
    RuntimeError: Cannot open file 'Groceries/positive/300px-Haw5_614.jpg'

    [CALL STACK]
    [0x7fca335727dc]                                                       + 0x2c7dc
    [0x7fca3358a46d]    Microsoft::MSR::CNTK::ImageDataDeserializer::ImageChunk::  GetSequence  (unsigned long,  std::vector<std::shared_ptr<Microsoft::MSR::CNTK::SequenceDataBase>,std::allocator<std::shared_ptr<Microsoft::MSR::CNTK::SequenceDataBase>>>&) + 0x2ad
    [0x7fca60ab6f3f]    Microsoft::MSR::CNTK::Bundler::BundlingChunk::  GetSequence  (unsigned long,  std::vector<std::shared_ptr<Microsoft::MSR::CNTK::SequenceDataBase>,std::allocator<std::shared_ptr<Microsoft::MSR::CNTK::SequenceDataBase>>>&) + 0x6f
    [0x7fca60aae437]    Microsoft::MSR::CNTK::BlockRandomizer::  LoadSequenceData  (unsigned long,  unsigned long,  Microsoft::MSR::CNTK::Sequences&,  bool) + 0x2d7
    [0x7fca60aaeb5c]    Microsoft::MSR::CNTK::BlockRandomizer::  GetNextSequences  (unsigned long,  unsigned long) + 0x8c
    [0x7fca337d3eeb]    Microsoft::MSR::CNTK::TransformController::  GetNextSequences  (unsigned long,  unsigned long) + 0x2b
    [0x7fca60acd2ac]    Microsoft::MSR::CNTK::SequencePacker::  ReadMinibatch  () + 0x3c
    [0x7fca60adfd0e]    Microsoft::MSR::CNTK::ReaderBase::  ReadMinibatch  () + 0xe
    [0x7fca60ac6d3f]    Microsoft::MSR::CNTK::ReaderShim<float>::  PrefetchMinibatch  (unsigned long) + 0xcf
    [0x7fca60ac7093]    std::_Function_handler<Microsoft::MSR::CNTK::ReaderShim<float>::PrefetchResult (),std::reference_wrapper<std::_Bind_simple<Microsoft::MSR::CNTK::ReaderShim<float>::StartEpoch(Microsoft::MSR::CNTK::EpochConfiguration const&,std::unordered_set<Microsoft::MSR::CNTK::InputStreamDescription,std::hash<Microsoft::MSR::CNTK::InputStreamDescription>,std::equal_to<Microsoft::MSR::CNTK::InputStreamDescription>,std::allocator<Microsoft::MSR::CNTK::InputStreamDescription>> const&)::{lambda()#3} ()>>>::  _M_invoke  (std::_Any_data const&) + 0x13
    [0x7fca60abe12e]    std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base,std::__future_base::_Result_base::_Deleter> (),std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<Microsoft::MSR::CNTK::ReaderShim<float>::PrefetchResult>,std::__future_base::_Result_base::_Deleter>,Microsoft::MSR::CNTK::ReaderShim<float>::PrefetchResult>>::  _M_invoke  (std::_Any_data const&) + 0x1e
    [0x7fca60aaf20e]    std::__future_base::_State_base::  _M_do_set  (std::function<std::unique_ptr<std::__future_base::_Result_base,std::__future_base::_Result_base::_Deleter> ()>&,  bool&) + 0x1e
    [0x7fca6c907a99]                                                       + 0xea99
    [0x7fca60abcf4c]                                                       + 0x93df4c
    [0x7fca60ac3f2d]    std::thread::_Impl<std::_Bind_simple<std::__future_base::_Async_state_impl<std::_Bind_simple<Microsoft::MSR::CNTK::ReaderShim<float>::StartEpoch(Microsoft::MSR::CNTK::EpochConfiguration const&,std::unordered_set<Microsoft::MSR::CNTK::InputStreamDescription,std::hash<Microsoft::MSR::CNTK::InputStreamDescription>,std::equal_to<Microsoft::MSR::CNTK::InputStreamDescription>,std::allocator<Microsoft::MSR::CNTK::InputStreamDescription>> const&)::{lambda()#3} ()>,Microsoft::MSR::CNTK::ReaderShim<float>::PrefetchResult>::_Async_state_impl(Microsoft::MSR::CNTK::ReaderShim<float>::StartEpoch(Microsoft::MSR::CNTK::EpochConfiguration const&,std::unordered_set<Microsoft::MSR::CNTK::InputStreamDescription,std::hash<Microsoft::MSR::CNTK::InputStreamDescription>,std::equal_to<Microsoft::MSR::CNTK::InputStreamDescription>,std::allocator<Microsoft::MSR::CNTK::InputStreamDescription>> const&)::{lambda()#3} (&&)())::{lambda()#1} ()>>::  _M_run  () + 0x6d
    [0x7fca627bbc80]                                                       + 0xb8c80
    [0x7fca6c9006ba]                                                       + 0x76ba
    [0x7fca6bf2682d]    clone                                              + 0x6d