仅从格式为.tex的arXiv文章中提取正文

Question

我的数据集是由arXiv天体物理学文章组成的.tex文件，我只需要从文章正文中提取文本，而不需要从文章的任何其他部分（例如表格、图表、摘要、标题、脚注、致谢、引文等）。在

我一直在尝试Python3和tex2py，但是我很难获得一个干净的语料库，因为文件在标记上有所不同，文本在标签之间是分开的。在

我附上了一个SSCCE，一对样本Latex文件及其pdf文件，以及解析后的语料库。语料库显示了我的努力：部分和小节没有按顺序提取，文本在某些标签处断开，并且包含了一些表格和图表。在

代码：

import os
from tex2py import tex2py

corpus = open('corpus2.tex', 'a')

def parseFiles():
    """
    Parses downloaded document .tex files for word content.
    We are only interested in the article body, defined by /section tags.
    """

    for file in os.listdir("latex"):
        if file.endswith('.tex'):
            print('\nChecking ' + file + '...')
            with open("latex/" + file) as f:
                try:
                    toc = tex2py(f) # toc = tree of contents
                    # If file is a document, defined as having \begin{document}
                    if toc.source.document:
                        # Iterate over each section in document
                        for section in toc:
                            # Parse the section
                            getText(section)
                    else:
                        print(file + ' is not a document. Discarded.')
                except (EOFError, TypeError, UnicodeDecodeError): 
                    print('Error: ' + file + ' was not correctly formatted. Discarded.')



def getText(section):
    """
    Extracts text from given "section" node and any nested "subsection" nodes. 

    Parameters
    ----------
    section : list
        A "section" node in a .tex document 
    """

    # For each element within the section 
    for x in section:
        if hasattr(x.source, 'name'):
            # If it is a subsection or subsubsection, parse it
            if x.source.name == 'subsection' or x.source.name == 'subsubsection':
                corpus.write('\nSUBSECTION!!!!!!!!!!!!!\n')
                getText(x)
            # Avoid parsing past these sections
            elif x.source.name == 'acknowledgements' or x.source.name == 'appendix':
                return
        # If element is text, add it to corpus
        elif isinstance(x.source, str):
            # If element is inline math, worry about it later
            if x.source.startswith('$') and x.source.endswith('$'):
                continue
            corpus.write(str(x))
        # If element is 'RArg' labelled, e.g. \em for italic, add it to corpus
        elif type(x.source).__name__ is 'RArg':
            corpus.write(str(x.source))


if __name__ == '__main__':
    """Runs if script called on command line"""
    parseFiles()

其他链接：

• Sample .tex file 1及其{a3}
• Sample .tex file 2及其{a5}
• 结果corpus

我知道一个相关的问题（Programatically converting/parsing latex code to plain text），但似乎没有一个确切的答案。在

Answer 1

要从文档中获取所有文本，tree.descendants在这里会更加友好。这将按顺序输出所有文本。在

def getText(section):
    for token in section.descendants:
        if isinstance(token, str):
            corpus.write(str(x))

为了捕捉边缘案例，我写了一个稍微充实的版本。这包括检查上面列出的所有情况。在

^{pr2}$