长时间运行的循环会变慢吗?

2024-09-29 01:33:19 发布

您现在位置:Python中文网/ 问答频道 /正文

我在python中有一个generator函数,它以块的形式读取数据集,并在循环中生成每个块。 在循环的每次迭代中,块大小都是相同的,并且数据数组被覆盖。你知道吗

它开始时每0.3秒产生一次块,到第70次迭代时,速度会减慢到每3秒一次。 这是发电机:

def yield_chunks(self):
  # Loop over the chunks
  for j in range(self.ny_chunks):
    for i in range(self.nx_chunks):         
      dataset_no = 0
      arr = numpy.zeros([self.chunk_size_y, self.chunk_size_x, nInputs], numpy.dtype(numpy.int32))

      # Loop over the datasets we will read into a single 'chunk'
      for peril in datasets.dataset_cache.iterkeys():
        group = datasets.getDatasetGroup(peril)
        for return_period, dataset in group:
          dataset_no += 1

          # Compute the window of the dataset that falls into this chunk
          dataset_xoff, dataset_yoff, dataset_xsize, dataset_ysize = self.chunk_params(i, j)

          # Read the data
          data = dataset[0].ReadAsArray(dataset_xoff, dataset_yoff, dataset_xsize, dataset_ysize)

          # Compute the window of our chunk array that this data fits into
          chunk_xoff, chunk_yoff = self.window_params(dataset_xoff, dataset_yoff, dataset_xsize, dataset_ysize)

          # Add the data to the chunk array
          arr[chunk_yoff:(dataset_ysize+chunk_yoff), chunk_xoff:(dataset_xsize+chunk_xoff), dataset_no] = data

      # Once we have added data from all datasets to the chunk array, yield it
      yield arr

有没有可能内存在每个块之后都没有被正确释放,这会导致循环变慢?还有其他原因吗?你知道吗


Tags: thenoinselffordatadatasetchunks