如何对大型（200000多条记录）集合运行pymongo聚合查询？

Question

我需要为一个有200000多条数据记录的大型集合运行聚合查询。我想用pymongo来运行它。我在文件里试过首选的方法。在

pipeline = [...]

db.command('aggregate', 'statCollection', pipeline=pipeline_aggregate)

但这返回了一个错误消息pymongo.errors.OperationFailure: The 'cursor' option is required, except for aggregate with the explain argument。在

Answer 1

我用allowDiskUse选项解决了这个问题。这就是我的答案。在

pipeline_2 = [...]

db.command('aggregate', 'statCollection', pipeline=pipeline_2, allowDiskUse=True, cursor={})

Answer 2

来自aggregation的mongo文档

Changed in version 3.4: MongoDB 3.6 removes the use of aggregate command without the cursor option unless the command includes the explain option. Unless you include the explain option, you must specify the cursor option.

您可以将runCommand更改为aggregate pipeline，或者在命令中提供cursor

无cursor错误的runCommand

> db.runCommand({aggregate : "coin_infos", pipeline : [ {$match : {"coin_code" : "BTC"}}]})
{
    "ok" : 0,
    "errmsg" : "The 'cursor' option is required, except for aggregate with the explain argument",
    "code" : 9,
    "codeName" : "FailedToParse"
}

with cursor返回游标

^{pr2}$

with explain:true和no cursor返回执行计划

> db.runCommand({aggregate : "coin_infos", pipeline : [ {$match : {"coin_code" : "BTC"}}], explain : true})
{
    "stages" : [
        {
            "$cursor" : {
                "query" : {
                    "coin_code" : "BTC"
                },
                "queryPlanner" : {
                    "plannerVersion" : 1,
                    "namespace" : "bitcoin.coin_infos",
                    "indexFilterSet" : false,
                    "parsedQuery" : {
                        "coin_code" : {
                            "$eq" : "BTC"
                        }
                    },
                    "winningPlan" : {
                        "stage" : "COLLSCAN",
                        "filter" : {
                            "coin_code" : {
                                "$eq" : "BTC"
                            }
                        },
                        "direction" : "forward"
                    },
                    "rejectedPlans" : [ ]
                }
            }
        }
    ],
    "ok" : 1
}