从一个巨大的csv fi中计算唯一行的数量

Question

我有一个巨大的csv文件（约5-6gb）的大小，这是在蜂巢。有没有办法计算文件中存在的唯一行数？你知道吗

我对此一无所知。你知道吗

我需要将输出与另一个具有类似内容但唯一值的配置单元表进行比较。所以，基本上我需要找出不同的LINNE数。你知道吗

Answer 1

下面的逻辑是基于哈希的。它读取每一行的哈希值，而不是整行的哈希值，这使大小最小化。然后比较散列。对于相等的字符串，散列通常是相同的，很少有字符串会发生变化，所以读取实际行并比较实际字符串以确定。下面的内容也适用于大文件。你知道吗

from collections import Counter
input_file = r'input_file.txt'

# Main logic
# If hash is different then the contents are different
# If hash is same then the contents may be different


def count_with_index(values):
    '''
    Returns dict like key: (count, [indexes])
    '''
    result = {}
    for i, v in enumerate(values):
        count, indexes = result.get(v, (0, []))
        result[v] = (count + 1, indexes + [i])
    return result


def get_lines(fp, line_numbers):
    return (v for i, v in enumerate(fp) if i in line_numbers)


# Gets hashes of all lines
counter = count_with_index(map(hash, open(input_file)))

# Sums only the unique hashes
sum_of_unique_hash = sum((c for _, (c, _) in counter.items() if c == 1))

# Filters all non unique hashes
non_unique_hash = ((h, v) for h, (c, v) in counter.items() if c != 1)

total_sum = sum_of_unique_hash

# For all non unique hashes get the actual line and count
# One hash is picked per time. So memory is not consumed much.
for h, v in non_unique_hash:
    counter = Counter(get_lines(open(input_file), v))
    total_sum += sum(1 for k, v in counter.items())

print('Total number of unique lines is : ', total_sum)