相互作用病理处理池还有圆周率

Question

我有一个要计算的列表。我用

from pathos.multiprocessing import ProcessingPool
pool = ProcessingPool(nodes=7)
values = pool.map(helperFunction, someArgs)

helperFunction确实创建了一个名为Parameters的类，该类在与

^{pr2}$

到目前为止，还不错。helperFunction将根据Parameters对象进行一些计算，更改其某些属性，最后使用pickle存储这些属性。以下是执行保存的helper函数（来自不同模块）的相关摘录：

import pickle
import hashlib
import os
class cacheHelper():

    def __init__(self, fileName, attr=[], folder='../cache/'):
        self.folder = folder

        if len(attr) > 0:
            attr = self.attrToName(attr)
        else:
            attr = ''
        self.fileNameNaked = fileName
        self.fileName = fileName + attr

    def write(self, objects):
        with open(self.getFile(), 'wb') as output:
            for object in objects:
                pickle.dump(object, output, pickle.HIGHEST_PROTOCOL)

当它到达pickle.dump()时，它会引发一个很难调试的异常，因为调试器不会单步进入实际遇到该异常的工作进程。因此，我在转储发生之前创建了一个断点，并手动输入了该命令。输出如下：

>>> pickle.dump(objects[0], output, pickle.HIGHEST_PROTOCOL)
Traceback (most recent call last):
  File "/usr/local/anaconda2/envs/myenv2/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2885, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-1-4d2cbb7c63d1>", line 1, in <module>
    pickle.dump(objects[0], output, pickle.HIGHEST_PROTOCOL)
  File "/usr/local/anaconda2/envs/myenv2/lib/python2.7/pickle.py", line 1376, in dump
    Pickler(file, protocol).dump(obj)
  File "/usr/local/anaconda2/envs/myenv2/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/local/anaconda2/envs/myenv2/lib/python2.7/pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "/usr/local/anaconda2/envs/myenv2/lib/python2.7/pickle.py", line 396, in save_reduce
    save(cls)
  File "/usr/local/anaconda2/envs/myenv2/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/local/anaconda2/envs/myenv2/lib/python2.7/site-packages/dill/dill.py", line 1203, in save_type
    StockPickler.save_global(pickler, obj)
  File "/usr/local/anaconda2/envs/myenv2/lib/python2.7/pickle.py", line 754, in save_global
    (obj, module, name))
PicklingError: Can't pickle <class '__main__.Parameters'>: it's not found as __main__.Parameters

奇怪的是，当我不并行化时，也就是说，手动循环helperFunction，这种情况就不会发生。我很确定我打开的是右Parameters（而不是父类）。在

我知道在没有可复制的例子的情况下调试是很困难的，我不希望在这方面有任何解决方案。也许更普遍的问题是：

当通过另一个模块并行化使用pickle.dump()的代码时，需要注意什么？

Answer 1

直接来自Pythondocs。在

12.1.4. What can be pickled and unpickled? The following types can be pickled:

• None, True, and False
• integers, floating point numbers, complex
• strings, bytes, bytearrays
• tuples, lists, sets, and
• dictionaries containing only picklable objects functions defined at the top level of a module (using def, not lambda)
• built-in functions defined at the top level of a module
• classes that are defined at the top level of a module
• instances of such classes whose __dict__ or the result of calling __getstate__() is picklable (see section Pickling Class Instances for details).

其他的都不能腌制。在您的例子中，虽然很难说给定代码的摘要，但我认为问题是类Parameters没有在模块的顶层定义，因此它的实例不能被pickle。在

使用pathos.multiprocessing（或其积极开发的forkmultiprocess）而不是内置的multiprocessing，这是为了避免{}，因为有太多的东西是后者无法转储的。pathos.multiprocessing和multiprocess使用dill而不是{}。如果您想调试一个worker，可以使用trace。在

注意正如Mike McKerns（关于multiprocess的主要贡献者）正确地注意到，有些情况下即使{}也无法处理，尽管很难在这方面制定一些通用规则。在