Python多处理：所有进程在5秒钟内完成，但程序需要额外10秒钟才能返回读取主脚本

Question

我最近开始学习Python多处理。 每次，在所有进程完成之后，python都需要额外的一秒钟来关闭一个进程（因此，如果打开10个进程，则需要10秒），然后返回脚本的其余部分。因此问题在于，一个简单的.sleep（）程序使用多处理比运行普通的线性程序需要更多的时间。 代码：

def do_stuff(seconds):
    print("Sleeping for {} seconds...".format(seconds))
    time.sleep(seconds)
    print("Done sleeping... yawn...")



if __name__ == "__main__":
    start = time.perf_counter()

    process_list = []
    for _ in range(10):
        p = multiprocessing.Process(target=do_stuff, args=[5])
        p.start()
        process_list.append(p)

    for process in process_list:
        process.join()


    end = time.perf_counter()
    print(end-start)

输出如下所示：

Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...

到目前为止，程序花了大约5秒钟完成所有10个过程

15.2574748

但是，它会再等待10秒来完成代码，并返回程序完成所需的时间 我如何回避这个问题？谢谢

Answer 1

你的密码似乎没问题。但是，如果它不适用于您的操作系统，您可以尝试定义希望同时运行的最大线程数。在下面的示例中，它被设置为10

import time
import multiprocessing

def do_stuff(seconds):
    print("Sleeping for {} seconds...".format(seconds))
    time.sleep(seconds)
    print("Done sleeping... yawn...")

def chunks(l, n):
    for i in range(0, len(l), n):
        yield l[i:i + n]    

numberOfThreads = 10

if __name__ == "__main__":
    start = time.perf_counter()

    process_list = []
    for _ in range(10):
        p = multiprocessing.Process(target=do_stuff, args=[5])
        process_list.append(p)

    for i in chunks(process_list,numberOfThreads):
        for j in i:
            j.start()
        for j in i:
            j.join()


    end = time.perf_counter()
    print(end-start)

结果:

Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
5.6765913999999995