避免或处理“BadRequestError:请求的查询已过期。”？

Question

我使用链式延迟任务和查询游标在appengine中循环数据。Python2.7，使用db（而不是ndb）。E、 g

def loop_assets(cursor = None):

  try:

     assets = models.Asset.all().order('-size')

     if cursor:
        assets.with_cursor(cursor)

     for asset in assets.run():

        if asset.is_special():
           asset.yay = True
           asset.put()

  except db.Timeout:
     cursor = assets.cursor()
     deferred.defer(loop_assets, cursor = cursor,  _countdown = 3, _target = version, _retry_options = dont_retry)
     return

这总共运行了约75分钟（每个任务持续~1分钟），然后引发了此异常：

BadRequestError: The requested query has expired. Please restart it with the last cursor to read more results.

阅读the docs，唯一说明的原因是：

New App Engine releases may change internal implementation details, invalidating cursors that depend on them. If an application attempts to use a cursor that is no longer valid, the Datastore raises a BadRequestError exception.

所以也许这就是事实，但我第一次尝试这种技术时，我遇到了一个“内部实现的变化”（除非经常发生）。在

这还有其他解释吗？ 有没有办法重新设计我的代码来避免这种情况？在

如果没有，我认为唯一的解决方案是标记哪些资产已被处理，然后向查询添加一个额外的过滤器以排除这些资产，然后在每次进程结束时手动重新启动该进程。在

作为参考，this question也提出了类似的问题，但公认的答案是“使用光标”，我已经在做了，所以不可能是同一个问题。在

Answer 1

您可能需要查看AppEngine MapReduce

MapReduce is a programming model for processing large amounts of data in a parallel and distributed fashion. It is useful for large, long-running jobs that cannot be handled within the scope of a single request, tasks like:

• Analyzing application logs
• Aggregating related data from external sources
• Transforming data from one format to another
• Exporting data for external analysis