从URL加载非常大的json文件时出现内存错误

Question

我在其他类似的问题上看到他们使用像ijson这样的库。但我似乎不知道如何用它来解决这个问题。你知道吗

这是我目前掌握的代码：

f = urllib.request.urlopen("https://data.medicaid.gov/resource/4qik-skk9.json?$limit=646259")
stuff = ijson.items(f,"")
for items in stuff:
print(items)

json结构如下所示：

[
    {
        "package_size_code": "60",
        "fda_ther_equiv_code": "NR",
        "fda_application_number": "204153",
        "clotting_factor_indicator": "N",
        "year": "2018",
        "fda_product_name": "LUZU Cream 1% 60gm",
        "labeler_name": "MEDICIS DERMATOLOGICS, INC.",
        "ndc": "99207085060",
        "product_code": "0850",
        "unit_type": "GM",
        "fda_approval_date": "2013-11-14T00:00:00",
        "market_date": "2014-03-14T00:00:00",
        "pediatric_indicator": "N",
        "package_size_intro_date": "2014-03-14T00:00:00",
        "units_per_pkg_size": "60000",
        "labeler_code": "99207",
        "desi_indicator": "1",
        "drug_category": "S",
        "quarter": "3",
        "cod_status": "3"
    },
    {
        "package_size_code": "60",
        "fda_ther_equiv_code": "AB",
        "fda_application_number": "21758",
        "clotting_factor_indicator": "N",
        "year": "2018",
        "fda_product_name": "VANOS CREAM .1%",
        "labeler_name": "MEDICIS DERMATOLOGICS, INC.",
        "ndc": "99207052560",
        "product_code": "0525",
        "unit_type": "GM",
        "fda_approval_date": "2005-02-11T00:00:00",
        "market_date": "2005-02-21T00:00:00",
        "pediatric_indicator": "N",
        "package_size_intro_date": "2005-02-21T00:00:00",
        "units_per_pkg_size": "60000",
        "labeler_code": "99207",
        "desi_indicator": "1",
        "drug_category": "I",
        "quarter": "3",
        "cod_status": "3"
    },
.
.
.
.
]

我要做的是得到所有的结果并对它们应用一些过滤器来获得一个值。例如，我试图从所有条目中获取年的最大值。要做到这一点，我想我需要阅读所有的数据。你知道吗

这似乎解决了内存问题：

parser = ijson.parse(urllib.request.urlopen('https://data.medicaid.gov/resource/4qik-skk9.json?$limit=646259'))
        for prefix, event, value  in parser:
            #print(prefix)
            print(event)
            #print(value)

数据出来的不太整齐，但正在进步。你知道吗