我正在为Luigi任务构建一个包装器,我遇到了一个问题,^{type
时,它不能被选择。在
下面的代码或多或少就是我用来开发动态类的代码。在
class TaskWrapper(object):
'''Luigi Spark Factory from the provided JobClass
Args:
JobClass(ScrubbedClass): The job to wrap
options: Options as passed into the JobClass
'''
def __new__(self, JobClass, **options):
# Validate we have a good job
valid_classes = (
ScrubbedClass01,
# ScrubbedClass02,
# ScrubbedClass03,
)
if any(vc == JobClass for vc in valid_classes) or not issubclass(JobClass, valid_classes):
raise TypeError('Job is not the correct class: {}'.format(JobClass))
# Build a luigi task class dynamically
luigi_identifier = 'Task'
job_name = JobClass.__name__
job_name = job_name.replace('Pail', '')
if not job_name.endswith(luigi_identifier):
job_name += luigi_identifier
LuigiTask = type(job_name, (PySparkTask, ), {})
for k, v in options.items():
setattr(LuigiTask, k, luigi.Parameter())
def main(self, sc, *args):
job = JobClass(**options)
return job._run()
LuigiTask.main = main
return LuigiTask
但是,当我运行调用函数时,我得到PicklingError: Can't pickle <class 'abc.ScrubbedNameTask'>: attribute lookup abc.ScrubbedNameTask failed
。在
调用函数:
^{pr2}$
当使用
ABC
的元类动态创建类时,该模块将成为abc
,当工作线程试图查找任务时,它将转到抽象基类模块并尝试在那里找到它,但它当然不存在。在要解决这个问题,请确保luigi知道在哪里可以找到构建类的代码,方法是手动重置
__module__
变量。在将行改为:
据我所知,这只是Windows上的一个问题。在
相关问题 更多 >
编程相关推荐