我尝试使用XML minidom解析这个巨大的XML文档。虽然它在一个示例文件上运行得很好,但在尝试处理实际文件(大约400MB)时,它阻塞了系统。在
我尝试从codereview调整代码(它以流方式处理数据,而不是一次在内存中加载数据),对于我的xml文件,由于元素的嵌套性质,我很难隔离数据集。我以前处理过简单的XML文件,但没有处理像这样的内存密集型任务。在
这是正确的方法吗? 如何将库存和出版商标识与每本书相关联?这就是我计划最终将这两个表关联起来的方式。在
如有任何反馈,我们将不胜感激。在
在图书.xml在
<BookDatabase>
<BookHeader>
<Name>BookData</Name>
<BookUniverse>All</BookUniverse>
<AsOfDate>2010-05-02</AsOfDate>
<Version>1.1</Version>
</BookHeader>
<InventoryBody>
<Inventory ID="12">
<PublisherClass ID="34">
<Publisher>
<PublisherDetails>
<Name>Microsoft Press</Name>
<Type>Tech</Type>
<ID>7462</ID>
</PublisherDetails>
</Publisher>
</PublisherClass>
<BookList>
<Listing>
<BookListSummary>
<Date>2009-01-30</Date>
</BookListSummary>
<Book>
<BookDetail ID="67">
<BookName>Code Complete 2</BookName>
<Author>Steve McConnell</Author>
<Pages>960</Pages>
<ISBN>0735619670</ISBN>
</BookDetail>
<BookDetail ID="78">
<BookName>Application Architecture Guide 2</BookName>
<Author>Microsoft Team</Author>
<Pages>496</Pages>
<ISBN>073562710X</ISBN>
</BookDetail>
</Book>
</Listing>
</BookList>
</Inventory>
<Inventory ID="64">
<PublisherClass ID="154">
<Publisher>
<PublisherDetails>
<Name>O'Reilly Media</Name>
<Type>Tech</Type>
<ID>7484</ID>
</PublisherDetails>
</Publisher>
</PublisherClass>
<BookList>
<Listing>
<BookListSummary>
<Date>2009-03-30</Date>
</BookListSummary>
<Book>
<BookDetail ID="98">
<BookName>Head First Design Patterns</BookName>
<Author>Kathy Sierra</Author>
<Pages>688</Pages>
<ISBN>0596007124</ISBN>
</BookDetail>
</Book>
</Listing>
</BookList>
</Inventory>
</InventoryBody>
</BookDatabase>
Python代码:
^{pr2}$Python输出:
$ python lxmletree_book.py
========> 0 <=======
========> 1 <=======
{'ID': '12', 'element': 'Inventory'} []
========> 2 <=======
{'ID': '34', 'element': 'PublisherClass'} []
========> 3 <=======
{'Name': <Element Name at 0x105140af0>, 'Type': <Element Type at 0x105140b40>, 'ID': <Element ID at 0x105140b90>, 'element': 'PublisherDetails'} []
========> 4 <=======
{'ID': None, 'element': 'PublisherDetails'} []
========> 5 <=======
{'ID': None, 'element': 'PublisherClass'} []
========> 6 <=======
{'ISBN': <Element ISBN at 0x105140eb0>, 'Name': <Element Name at 0x105140dc0>, 'Author': <Element Author at 0x105140e10>, 'ID': '67', 'element': 'BookDetail', 'Pages': <Element Pages at 0x105140e60>} []
========> 7 <=======
{'ID': None, 'element': 'BookDetail'} []
========> 8 <=======
{'ISBN': <Element ISBN at 0x1051460a0>, 'Name': <Element Name at 0x105140f50>, 'Author': <Element Author at 0x105140fa0>, 'ID': '78', 'element': 'BookDetail', 'Pages': <Element Pages at 0x105146050>} []
========> 9 <=======
{'ID': None, 'element': 'BookDetail'} []
========> 10 <=======
{'ID': None, 'element': 'Inventory'} []
========> 11 <=======
{'ID': '64', 'element': 'Inventory'} []
========> 12 <=======
{'ID': '154', 'element': 'PublisherClass'} []
========> 13 <=======
{'Name': <Element Name at 0x105146230>, 'Type': <Element Type at 0x105146280>, 'ID': <Element ID at 0x1051462d0>, 'element': 'PublisherDetails'} []
========> 14 <=======
{'ID': None, 'element': 'PublisherDetails'} []
========> 15 <=======
{'ID': None, 'element': 'PublisherClass'} []
========> 16 <=======
{'ISBN': <Element ISBN at 0x1051465f0>, 'Name': <Element Name at 0x105146500>, 'Author': <Element Author at 0x105146550>, 'ID': '98', 'element': 'BookDetail', 'Pages': <Element Pages at 0x1051465a0>} []
========> 17 <=======
{'ID': None, 'element': 'BookDetail'} []
========> 18 <=======
{'ID': None, 'element': 'Inventory'} []
========> 19 <=======
期望的输出(最终存储在MySQL中—现在是Python中的列表):
Publishers
InventoryID PublisherClassID Name Type ID
12 34 Microsoft Press Tech 7462
64 154 O'Reilly Media Tech 7484
Books
PublisherID BookDetailID Name Author Pages ISBN
7462 67 Code Complete 2 Steve McConnell 960 0735619670
7462 78 Application Architecture Guide 2 Microsoft Team 496 073562710X
7484 98 Head First Design Patterns Kathy Sierra 688 0596007124
因此,如果你想要快速而肮脏的东西,你只需记住最后一个被解析的发布者,并将其分配给每个bookdetail,直到找到下一个PublisherDetails。在提取bookdetail之后,向fast_iter2函数添加如下内容:
你可以试试这样的方法:
fast_iter2
。original ^{extract_paper_elements
)。fast_iter2
将两者混合在一起 没有可重复的代码。在etree.iterparse("book.xml", events=("end",), tag='Inventory')
中设置tag
参数,则处理函数extract_paper_elements
将只看到Inventory
元素。在xpath
方法进行挖掘 把所需的数据收集下来。在args
和kwargs
参数被添加到fast_iter
所以cursor
可以传递给extract_paper_elements
。在相关问题 更多 >
编程相关推荐