在Python(2.7)中,我试图在芹菜任务(芹菜3.1.17)中创建进程(使用多进程),但它给出了错误:
daemonic processes are not allowed to have children
通过谷歌搜索,我发现最新版本的台球修正了“错误”,但我有最新版本(3.3.0.20),错误仍在发生。我还试图在芹菜任务中实现this workaround,但它给出了相同的错误。
有人知道怎么做吗? 感谢您的帮助, 帕特里克
编辑:代码片段
任务:
from __future__ import absolute_import
from celery import shared_task
from embedder.models import Embedder
@shared_task
def embedder_update_task(embedder_id):
embedder = Embedder.objects.get(pk=embedder_id)
embedder.test()
人工测试函数(from here):
def sleepawhile(t):
print("Sleeping %i seconds..." % t)
time.sleep(t)
return t
def work(num_procs):
print("Creating %i (daemon) workers and jobs in child." % num_procs)
pool = mp.Pool(num_procs)
result = pool.map(sleepawhile,
[randint(1, 5) for x in range(num_procs)])
# The following is not really needed, since the (daemon) workers of the
# child's pool are killed when the child is terminated, but it's good
# practice to cleanup after ourselves anyway.
pool.close()
pool.join()
return result
def test(self):
print("Creating 5 (non-daemon) workers and jobs in main process.")
pool = MyPool(5)
result = pool.map(work, [randint(1, 5) for x in range(5)])
pool.close()
pool.join()
print(result)
Myreal函数:
import mulitprocessing as mp
def test(self):
self.init()
for saveindex in range(self.start_index,self.start_index+self.nsaves):
self.create_storage(saveindex)
# process creation:
procs = [mp.Process(name="Process-"+str(i),target=getattr(self,self.training_method),args=(saveindex,)) for i in range(self.nproc)]
for p in procs: p.start()
for p in procs: p.join()
print "End of task"
init函数定义了一个多处理数组和一个共享相同内存的对象,以便我的所有进程可以同时更新这个数组:
mp_arr = mp.Array(c.c_double, np.random.rand(1000000)) # example
self.V = numpy.frombuffer(mp_arr.get_obj()) #all the processes can update V
调用任务时生成错误:
[2015-06-04 09:47:46,659: INFO/MainProcess] Received task: embedder.tasks.embedder_update_task[09f8abae-649a-4abc-8381-bdf258d33dda]
[2015-06-04 09:47:47,674: WARNING/Worker-5] Creating 5 (non-daemon) workers and jobs in main process.
[2015-06-04 09:47:47,789: ERROR/MainProcess] Task embedder.tasks.embedder_update_task[09f8abae-649a-4abc-8381-bdf258d33dda] raised unexpected: AssertionError('daemonic processes are not allowed to have children',)
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/home/patrick/django/entite-tracker-master/entitetracker/embedder/tasks.py", line 21, in embedder_update_task
embedder.test()
File "/home/patrick/django/entite-tracker-master/entitetracker/embedder/models.py", line 475, in test
pool = MyPool(5)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 159, in __init__
self._repopulate_pool()
File "/usr/lib/python2.7/multiprocessing/pool.py", line 223, in _repopulate_pool
w.start()
File "/usr/lib/python2.7/multiprocessing/process.py", line 124, in start
'daemonic processes are not allowed to have children'
AssertionError: daemonic processes are not allowed to have children
当我使用芹菜4.2.0和Python3.6的多处理时,我得到了这个。 用台球解决了这个问题。
我更改了源代码
from multiprocessing import Process
到
from billiard.context import Process
解决了这个错误。
注意,输入源是
billiard.context
而不是billiard.process
我在django的芹菜任务中尝试调用多处理方法时遇到了类似的错误。我用台球代替了多处理
希望有帮助。
billiard
和multiprocessing
是不同的库-billiard
是芹菜项目自己的分支。您需要导入billiard
,并使用它而不是multiprocessing
然而,更好的答案可能是,您应该重构代码,以便生成更多的芹菜任务,而不是使用两种不同的方式分发您的工作。
你可以用芹菜canvas
我试图使您的代码的工作版本使用画布原语而不是多处理。然而,由于你的例子是非常人为的,所以要想想出一些有意义的东西并不容易。
更新:
以下是使用芹菜画布的真实代码翻译:
tasks.py
:models.py
:相关问题 更多 >
编程相关推荐