pythondocx lxml.etree.XMLSyntaxError:AttValue长度太长

Question

我正在编写一个程序来检查一堆.docx文件中是否存在一个单词（我们谈论的大约是2500个.docx文件）

下面是代码中有趣的部分：

for filename in directorylist:
    if filename.endswith(".docx"):
        i = Document(filename)

        print(filename)


        for destination in destinationlist:
            for paragraph in i.paragraphs:
                if destination in paragraph.text:
                    destinationcount[destination] = 1
                    break
                else:
                    destinationcount[destination] = 0
                    continue

        for destination in destinationcount:
            destinationcountnobool[destination] += destinationcount[destination]

    else:
        continue

现在，我知道你在想什么了，一般来说，这是一个非常混乱的循环和糟糕的编程，但这是一个快速而肮脏的工作，所以饶了我吧

下面是我得到的错误：

Traceback (most recent call last):
  File "ICrunchMeSomeFiles.py", line 27, in <module>
    i = Document(filename)
  File "C:\Users\User\Anaconda3\lib\site-packages\docx\api.py", line 25, in Document
    document_part = Package.open(docx).main_document_part
  File "C:\Users\User\Anaconda3\lib\site-packages\docx\opc\package.py", line 130, in open
    Unmarshaller.unmarshal(pkg_reader, package, PartFactory)
  File "C:\Users\User\Anaconda3\lib\site-packages\docx\opc\package.py", line 199, in unmarshal
    pkg_reader, package, part_factory
  File "C:\Users\User\Anaconda3\lib\site-packages\docx\opc\package.py", line 216, in _unmarshal_parts
    partname, content_type, reltype, blob, package
  File "C:\Users\User\Anaconda3\lib\site-packages\docx\opc\part.py", line 191, in __new__
    return PartClass.load(partname, content_type, blob, package)
  File "C:\Users\User\Anaconda3\lib\site-packages\docx\opc\part.py", line 231, in load
    element = parse_xml(blob)
  File "C:\Users\User\Anaconda3\lib\site-packages\docx\oxml\__init__.py", line 28, in parse_xml
    root_element = etree.fromstring(xml, oxml_parser)
  File "src\lxml\etree.pyx", line 3236, in lxml.etree.fromstring
  File "src\lxml\parser.pxi", line 1876, in lxml.etree._parseMemoryDocument
  File "src\lxml\parser.pxi", line 1764, in lxml.etree._parseDoc
  File "src\lxml\parser.pxi", line 1127, in lxml.etree._BaseParser._parseDoc
  File "src\lxml\parser.pxi", line 601, in lxml.etree._ParserContext._handleParseResultDoc
  File "src\lxml\parser.pxi", line 711, in lxml.etree._handleParseResult
  File "src\lxml\parser.pxi", line 640, in lxml.etree._raiseParseError
  File "<string>", line 2
lxml.etree.XMLSyntaxError: AttValue length too long, line 2, column 11011745

该程序适用于较小的样本，因此我假设这是内存问题。非常感谢您的帮助

编辑：应该早一点这样做，但是已经用导致错误的整个代码片段更新了帖子

import csv
from docx import Document
from collections import Counter
import os

directorylist = os.listdir(os.getcwd()) # Set directory here
destinationcount = Counter()
destinationcountnobool = Counter()
destinationlist = ["test1", "test2", "test3", "test4", "test5"]
print(directorylist)

for filename in directorylist:
    if filename.endswith(".docx"):
        i = Document(filename)    


        for destination in destinationlist:
            for paragraph in i.paragraphs:
                if destination in paragraph.text:
                    destinationcount[destination] = 1
                    break
                else:
                    destinationcount[destination] = 0
                    continue

        for destination in destinationcount:
            destinationcountnobool[destination] += destinationcount[destination]

    else:
        continue


for d in destinationcountnobool:
    print(d + " : " + str(destinationcountnobool[d]))

更新：我研究这个问题已经有一段时间了……在遇到同样的错误之前，python似乎只能处理118个文件。

更新：解决了！我想有点……我已经发布了我的答案

Answer 1

我认为可以解决你的问题：

import os
from collections import defaultdict

directorylist = os.listdir('.')


destinationlist = []
destinationcount = defaultdict(int)


def return_each_paragraph_from_file(directory):
    for filename in directory:
        if filename.endswith(".docx"):
            i = Document(filename)
            for paragraph in i.paragraphs:
                yield paragraph


if __name__ == "main":
    for p in return_each_paragraph_from_file(directorylist):
        for destination in destinationlist:
            destinationcount[destination] += 1 if destination in p.text else destinationcount[destination]

Answer 2

好吧，经过一段合理的扣除。我终于弄明白发生了什么事。这似乎是一个记忆问题

我简单地接受了一个假设，即.docx文件具有某种不可知的特性，导致程序崩溃。但仔细检查后，我得出结论，这是不可能的，因为文件非常相似

下一步行动是复制这个项目，并将样本量减少到只有失败前处理的数量。电话号码是118

我试着对一个文件进行加减，但我似乎总是遇到118的硬限制。第119个文件总是导致它崩溃。解决办法很简单

for filename in directorylist:
    if filename.endswith(".docx"):
        i = Document(filename)    


        for destination in destinationlist:
            for paragraph in i.paragraphs:
                if destination in paragraph.text:
                    destinationcount[destination] = 1
                    break
                else:
                    destinationcount[destination] = 0
                    continue
        del i

        for destination in destinationcount:
            destinationcountnobool[destination] += destinationcount[destination]

我只需要删除对象以释放内存。而且，with似乎不适用于python docx

呸。。。就这样

编辑：仍然很好奇为什么它在第119个文件之前处理得很好。如果有人能启发我，我会很高兴的