在Python中解析大型XML文件时遇到内存错误

Question

我的XML文件如下所示：

<root>
<group from="1", to="100">
    <link target="1"/>
    ...
    <link target="100"/>
</group>
...
</root>

我有6000个元素和5个元素。我想要一个以元组（from，to）为键和<link>starget属性列表的字典，但是我得到一个内存错误，代码如下：

^{pr2}$

Answer 1

尝试以下代码：

在lxml.etree在

import lxml.etree
from gzip import open as gopen

class GroupDictTarget(object):
    def __init__(self, d):
        self.d = d
    def start(self, tag, attrib):
        if tag == 'group':
            self.group = self.d[attrib['from'], attrib['to']] = []
        elif tag == 'link':
            self.group.append(attrib['target'])
    def close(self):
        pass

def extractTargets(fin):
    with gopen(fin) as xml:
        targets = {}
        parser = lxml.etree.XMLParser(target=GroupDictTarget(targets))
        lxml.etree.parse(xml, parser)
        return targets

在xml.parsers.expat在

^{pr2}$

在xml.sax文件在

import xml.sax
from gzip import open as gopen

class GroupDictTarget(object):
    # Same as above

def extractTargets(fin):
    targets = {}
    handler = xml.sax.handler.ContentHandler()
    handler.startElement = GroupDictTarget(targets).start
    with gopen(fin) as f:
        xml.sax.parse(f, handler)
    return targets

Answer 2

我今天也遇到了同样的问题，在我删除了“tag”参数之后，它就起作用了：

context = etree.iterparse(xml)

for event, elem in context:
        if elem.tag = "group":
            targets[(elem.get("from"), elem.get("to"))] = elem.xpath("link/@target")
        elem.clear()

        while elem.getprevious() is not None:
            del elem.getparent()[0]
    del context