遍历巨大的文本文件：使用python读取两个重复出现的模式之间的块

Question

我需要通过一个巨大的（20GB，太大的内存）文本文件的生物序列数据库（GeneBank），并提取相同的信息为每个数据库条目。每个条目以LOCUS XYZ some more text行开始，以//行结束。例如：

LOCUS 123 some more text many lines of some more text many lines of some more text many lines of some more text // LOCUS 231 some more text many lines of some more text many lines of some more text many lines of some more text // LOCUS 312 some more text many lines of some more text many lines of some more text many lines of some more text //

现在，有没有一种方法可以告诉python迭代地将该文件中相应的3个块读取到某个变量var中。更准确地说：

迭代1:var=

LOCUS 123 some more text many lines of some more text many lines of some more text many lines of some more text //

迭代2:var=

LOCUS 231 some more text many lines of some more text many lines of some more text many lines of some more text //

迭代3:var=

LOCUS 312 some more text many lines of some more text many lines of some more text many lines of some more text //

预祝您节日愉快，万事如意

Answer 1

假设我们有以下文本文件：

LOCUS 421 bla bla ba
Lorem ipsum dolor sit amet, 
consectetur adipiscing elit. 
Duis eu erat orci. Quisque 
nec augue ultricies, dignissim 
neque id, feugiat risus.
//
LOCUS 421 blabla
Nullam pulvinar quis ante
at condimentum.
//

我们可以做：

is_processing = True

pf = open("somefile.txt", "r")

# Handles chunks
while True:
    first_chunk_line = True
    chunk_lines = []

    # Handles one chunk
    while True:
        data_line = pf.readline()

        # detect the end of the file
        if data_line == '':
            is_processing = False
            break

        # Detect first line
        if first_chunk_line:
            if "LOCUS" not in data_line:
                raise Exception("Data file is malformed!")

            first_chunk_line = False
            continue  # don't process the line
        # Detect end of locus / chunk
        if data_line.strip() == "//":
            break

        # if it is neither a first line, and end line nor the end of the file
        # then it must be a chunk line holding precious DNA information
        chunk_lines.append(data_line)

    # end the while loop
    if not is_processing:
        break

    # do something with one chunk lines
    print(chunk_lines)