python2和python3多处理。进程问题

Question

我试图了解在多处理模块中python2和python3之间发生了什么变化。 在python2上运行此代码就像一个符咒：

def RunPrice(items, price):
    print("There is %s items, price is: %s" % (items, price))

def GetTargetItemsAndPrice(cursor):
    res = cursor.execute("SELECT DISTINCT items, price FROM SELLS")
    threads = []
    for row in res.fetchall():
        p = multiprocessing.Process(target=RunPrice, args=(row[0],row[1]))
        threads.append(p)
        p.start()
    for proc in threads:
        proc.join()

假设在销售中有2000个条目需要处理。在python2上，此脚本按预期运行并退出。 在python3上，我得到一个：

  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 69, in _launch
    child_r, parent_w = os.pipe()
OSError: [Errno 24] Too many open files

知道Python2和Python3之间发生了什么吗

Answer 1

我假设您实际的RunPrice函数比您显示的要占用更多的CPU。否则，这将不是多处理的好选择。如果^ {CD1> }是CPU密集型的，并且不放弃CPU等待I/O完成，那么当您认为创建过程不是一种特别廉价的操作时，拥有比您拥有的CPU核数更多的进程池将是不利的。（虽然肯定没有在Windows上运行时那么昂贵）

from multiprocessing import Pool

def RunPrice(items, price):
    print("There is %s items, price is: %s" % (items, price))

def GetTargetItemsAndPrice(cursor):
    res = cursor.execute("SELECT DISTINCT items, price FROM SELLS")
    rows = res.fetchall()
    MAX_POOL_SIZE = 1024
    # if RunPrice is very CPU-intensive, it may not pay to have a pool size
    # greater than the number of CPU cores you have. In that case:
    #from multiprocessing import cpu_count
    #MAX_POOL_SIZE = cpu_count()
    pool_size = min(MAX_POOL_SIZE, len(rows))
    with Pool(pool_size) as pool:
        # return values from RunPrice:
        results = pool.starmap(RunPrice, [(row[0], row[1]) for row in rows])