在多进程循环中访问全局变量

Question

我不知道发生了什么？！这很奇怪，你能解释一下原因并修复它吗？ 我试图在2个循环中运行多处理，为了减少计算量，我需要计算一些全局变量，即'ga'和'gb'，但在进入多处理的函数'sirgr'中，ga和gb会扭曲。ga到（循环1的最后一个范围减去1加100，我的意思是107）和gb（循环2的最后一个范围减去1加100，我的意思是123），那么发生了什么？{}也没有任何帮助。 我知道ofc，我可以将其发送到函数，但我需要全局

import multiprocessing
def sirgr(row,col):
    global ga,gb
    print('sirgr',ga,gb)
    return ((row,col),(ga,gb))
def main():
    print('mainf',ga,gb)
    if __name__=="__main__":
        with multiprocessing.Pool() as pool:
            print('ino',tupleinput)
            output = list(pool.starmap(sirgr,tupleinput))
            print('out',output)
process_num=2
tupleinput=[]
for row in range(8):
    ga=row+100
    for col in range(20,24):
        gb=col+100
        print("r",row,'c',col)
        print('mainit',ga,gb)
        tupleinput.append((row,col))
        if len(tupleinput)==process_num:
            main()
            tupleinput=[]

结果:

#############
r 0 c 20
mainit 100 120
r 0 c 21
mainit 100 121
mainf 100 121
ino [(0, 20), (0, 21)]
out [((0, 20), (107, 123)), ((0, 21), (107, 123))]
#############
r 0 c 22
mainit 100 122
r 0 c 23
mainit 100 123
mainf 100 123
ino [(0, 22), (0, 23)]
out [((0, 22), (107, 123)), ((0, 23), (107, 123))]
....
#############
r 7 c 20
mainit 107 120
r 7 c 21
mainit 107 121
mainf 107 121
ino [(7, 20), (7, 21)]
out [((7, 20), (107, 123)), ((7, 21), (107, 123))]
#############
r 7 c 22
mainit 107 122
r 7 c 23
mainit 107 123
mainf 107 123
ino [(7, 22), (7, 23)]
out [((7, 22), (107, 123)), ((7, 23), (107, 123))]

例如，在r 0 c 21之后的输出中，输出预期为

out [((0, 20), (100, 121)), ((0, 21), (100, 121))]

但事实确实如此

out [(({row}, {col-1}), (107, 123)), (({row}, {col}), (107, 123))]用于所有输出 但我应该是

out [(({row}, {col-1}), ({row}, {col+100})), (({row}, {col}), {col+100})]

Answer 1

您没有提供q函数，所以我创建了自己的函数

请参阅我的评论：无法修改主进程中的常规全局变量，并希望它们在子进程的addrss空间中显示为已更改。如果坚持使用全局变量，则这些变量必须是管理的，共享的变量。必须通过它们的value属性访问它们的实际值

注意：您的代码一次只调用两个元组的starmap，这并不是最有效的处理方法，当然，除非您只有两个CPU核，但即使这样，还是最好将所有任务排队，一旦空闲时间减少，就调用starmap。由于您创建的池没有参数，因此它将创建一个池大小等于您拥有的CPU内核数，很可能大于两个。因此，您应该利用您拥有的任何CPU

import multiprocessing

def q(*args):
    args = [str(arg) for arg in args]
    print(', '.join(args))

def init_pool(globala, globalb):
    global ga, gb
    ga = globala
    gb = globalb

def sirgr(row, col):
    q('sirgr', ga.value, gb.value)
    return ((row,col),(ga.value, gb.value))

def main(pool, tupleinput):
    q('mainf', ga.value, gb.value)
    q('ino',tupleinput)
    output = pool.starmap(sirgr, tupleinput)
    q('out',output)

if __name__ == '__main__':
    with multiprocessing.Manager() as manager:
        ga = manager.Value('i', 0)
        gb = manager.Value('i', 0)
        pool = multiprocessing.Pool(initializer=init_pool, initargs=(ga, gb))
        tupleinput=[]
        for row in range(8):
            ga.value = row+100
            for col in range(20,24):
                gb.value = col+100
                q("r", row, 'c', col)
                q('mainit',ga.value, gb.value)
                tupleinput.append((row, col))
        main(pool, tupleinput)

印刷品：

r, 0, c, 20
mainit, 100, 120
r, 0, c, 21
mainit, 100, 121
r, 0, c, 22
mainit, 100, 122
r, 0, c, 23
mainit, 100, 123
r, 1, c, 20
mainit, 101, 120
r, 1, c, 21
mainit, 101, 121
r, 1, c, 22
mainit, 101, 122
r, 1, c, 23
mainit, 101, 123
r, 2, c, 20
mainit, 102, 120
r, 2, c, 21
mainit, 102, 121
r, 2, c, 22
mainit, 102, 122
r, 2, c, 23
mainit, 102, 123
r, 3, c, 20
mainit, 103, 120
r, 3, c, 21
mainit, 103, 121
r, 3, c, 22
mainit, 103, 122
r, 3, c, 23
mainit, 103, 123
r, 4, c, 20
mainit, 104, 120
r, 4, c, 21
mainit, 104, 121
r, 4, c, 22
mainit, 104, 122
r, 4, c, 23
mainit, 104, 123
r, 5, c, 20
mainit, 105, 120
r, 5, c, 21
mainit, 105, 121
r, 5, c, 22
mainit, 105, 122
r, 5, c, 23
mainit, 105, 123
r, 6, c, 20
mainit, 106, 120
r, 6, c, 21
mainit, 106, 121
r, 6, c, 22
mainit, 106, 122
r, 6, c, 23
mainit, 106, 123
r, 7, c, 20
mainit, 107, 120
r, 7, c, 21
mainit, 107, 121
r, 7, c, 22
mainit, 107, 122
r, 7, c, 23
mainit, 107, 123
mainf, 107, 123
ino, [(0, 20), (0, 21), (0, 22), (0, 23), (1, 20), (1, 21), (1, 22), (1, 23), (2, 20), (2, 21), (2, 22), (2, 23), (3, 20), (3, 21), (3, 22), (3, 23), (4, 20), (4, 21), (4, 22), (4, 23), (5, 20), (5, 21), (5, 22), (5, 23), (6, 20), (6, 21), (6, 22), (6, 23), (7, 20), (7, 21), (7, 22), (7, 23)]
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
sirgr, 107, 123
out, [((0, 20), (107, 123)), ((0, 21), (107, 123)), ((0, 22), (107, 123)), ((0, 23), (107, 123)), ((1, 20), (107, 123)), ((1, 21), (107, 123)), ((1, 22), (107, 123)), ((1, 23), (107, 123)), ((2, 20), (107, 123)), ((2, 21), (107, 123)), ((2, 22), (107, 123)), ((2, 23), (107, 123)), ((3, 20), (107, 123)), ((3, 21), (107, 123)), ((3, 22), (107, 123)), ((3, 23), (107, 123)), ((4, 20), (107, 123)), ((4, 21), (107, 123)), ((4, 22), (107, 123)), ((4, 23), (107, 123)), ((5, 20), (107, 123)), ((5, 21), (107, 123)), ((5, 22), (107, 123)), ((5, 23), (107, 123)), ((6, 20), (107, 123)), ((6, 21), (107, 123)), ((6, 22), (107, 123)), ((6, 23), (107, 123)), ((7, 20), (107, 123)), ((7, 21), (107, 123)), ((7, 22), (107, 123)), ((7, 23), (107, 123))]

更新

如果确实希望一次提交两个任务，则（假设要提交的任务总数是2的倍数）：

if __name__ == '__main__':
    with multiprocessing.Manager() as manager:
        ga = manager.Value('i', 0)
        gb = manager.Value('i', 0)
        pool = multiprocessing.Pool(2, initializer=init_pool, initargs=(ga, gb))
        tupleinput=[]
        for row in range(8):
            ga.value = row+100
            for col in range(20,24):
                gb.value = col+100
                q("r", row, 'c', col)
                q('mainit',ga.value, gb.value)
                tupleinput.append((row, col))
                if len(tupleinput) == 2:
                    main(pool, tupleinput)
                    tupleinput = []

更新2

我只是碰巧重新访问了它，看到我正在为ga和gb使用托管的Value实例，这两个实例都可以工作，但比共享内存Value实例的开销要大，对于这个问题，您只需要这些实例：

if __name__ == '__main__':
    ga = multiprocessing.Value('i', 0, Lock=False) # no synchronization required
    gb = multiprocessing.Value('i', 0, Lock=False) # no synchronization required
    pool = multiprocessing.Pool(2, initializer=init_pool, initargs=(ga, gb))
    ... # etc.