如何在python中对函数进行多处理

Question

我的代码：

for batchList in all_List:
    result = getBatch_filter(batchList, entTotal, tripleList, neg_rate)

def getBatch_filter(batchList, entTotal, tripleList, neg_rate):
    result = []
    for item in batchList:
        a = corrupt_a(item, entTotal, tripleList, neg_rate)
        b = corrupt_b(item, entTotal, tripleList, neg_rate)
        result += a
        result += b
    return result

其中，batchList、a、b是嵌套列表，如：[[1,2,3],[3,4,1,4,6],[6,9,10,11]...] 我想对函数getBatch_filter中的for循环进行多处理。我尝试了Pool，但它运行整个文件而不是函数。任何帮助都将不胜感激

Answer 1

请参见此示例：

from multiprocessing import Pool
def heavy_func(key):
    #do some heavy computation on each key 
    output = key**2
    return key, output 

output_data ={}     #< this dict will store the results
keys = [1,5,7,8,10] #< compute heavy_func over all the values of keys
with Pool(processes=40) as pool:
    for i in pool.imap_unordered(heavy_func, keys):
        output_data[i[0]] = i[1]

因此，在您的情况下，您将执行以下操作：

from multiprocessing import Pool
# assuming  entTotal, tripleList, neg_rate are globals and defined here
def getBatch_filter(batchList):
    result = []
    for item in batchList:
        a = corrupt_a(item, entTotal, tripleList, neg_rate)
        b = corrupt_b(item, entTotal, tripleList, neg_rate)
        result += a
        result += b
    return batchList, result


output_data ={}     #< this dict will store the results
keys = all_List #< compute heavy_func over all the values of keys
with Pool(processes=40) as pool:
    for i in pool.imap_unordered(getBatch_filter, keys):
        output_data[i[0]] = i[1]