不带映射的多处理附加到池

Question

我有一个函数将一组字符串加密到md5，在它里面，我创建了一个池

Main.py

config = ConfigParser()
config.read("config.ini")

possibleCharacters = "abcd"

def mapped_loop_digit(args):
     loop_digit(*args, is_pool=True)

def loop_digit(current_str, place, strings, hashes, is_outer=False, is_pool=False):
     if place == config.getint("string_creation", "length_for_new_process"):
         current_strings = list()

     for character in possibleCharacters:
         current_str[place] = character

         if is_outer and config.getboolean("development", "minor_logging"):
             print("Outer character maker at", possibleCharacters.index(character) + 1, "in", len(possibleCharacters))

         elif is_pool and config.getboolean("development", "pool_minor_logging"):
             print("Outest in pool character maker for process", multiprocessing.current_process()._identity[0],
              "at", possibleCharacters.index(character) + 1, "in", len(possibleCharacters), "with character as",
              str(character) + ". Current string is", current_str)

    if place == 0:
        string = "".join(_character for _character in current_str)
        hashes.append(hashlib.md5(string.encode()).hexdigest())
        strings.append(string)

    elif place == config.getint("string_creation", "length_for_new_process"):
        current_strings.append(current_str.copy())

    else:
        loop_digit(current_str, place - 1, strings, hashes)

     if place == config.getint("string_creation", "length_for_new_process"):
         args = list()
         print("Starting a new pool")
         for string in current_strings:
             args.append([string, place - 1, strings, hashes])

         with multiprocessing.Pool(processes=config.getint("string_creation", "processes")) as pool:
             pool.map(mapped_loop_digit, args)
             pool.close()
             pool.join()

manager = multiprocessing.Manager()
all_strings = manager.list("")
all_hashes = manager.list("")

loop_digit(["", "", "", ""], 4 - 1, all_strings, all_hashes, is_outer=True)

config.ini

[development]
minor_logging = 1
pool_minor_logging = 1

[string_creation]
processes = 3
length_for_new_process = 3

目前我有一个名为CurrutyStand的列表，并在程序中间附加到它，然后在结尾，我循环它并创建一个参数列表，然后将它映射到一个单独的函数，然后再次运行原来的函数。有没有一种更简单的方法来实现这一点，这样我就可以直接附加到池中而不是列表中

Answer 1

如果将Pool创建为

pool = multiprocessing.Pool(5)

如果没有pool.close(){}，则可以在不同的位置（在不同的函数中）多次使用pool

如果使用map_async()而不是map()，则不必等待进程结束，您可以使用next map_async()添加更多进程，pool将同时管理所有进程

您还可以使用apply_async将单个进程添加到现有的pool

因为map_async和apply_async不等待进程结束，所以您必须在退出程序之前使用wait()对其进行控制

it1 = pool.map_async(...)
it2 = pool.map_async(...)
it3 = pool.apply_async(...)

# ... code ...

it1.wait()
it2.wait()
it3.wait()

或者你必须在最后使用（两者）

pool.close()
pool.join()

若您不使用它，那个么程序可能会在进程完成之前退出，并终止它们

---

最小工作示例

import multiprocessing
import time

def fun(number):
    for x in range(3):
        time.sleep(.2)
        print(number, 'loop:', x)

if __name__ == '__main__':
    pool = multiprocessing.Pool(2)

    print("map [1,2,3]")
    it1 = pool.map_async(fun, [1,2,3])

    print("map ['A', 'B', 'C']")
    it2 = pool.map_async(fun, ['A', 'B', 'C'])

    print("single work X")
    it3 = pool.apply_async(fun, 'X')

    print("single work Y")
    it4 = pool.apply_async(fun, 'Y')

    # wait for the end of processes

    print('wait for the end of processes')

    #it1.wait()
    #it2.wait()
    #it3.wait()
    #it4.wait()

    pool.close()
    pool.join()

    print('exit')