PySpark UnpicklingError:NEWOBJ类参数具有空tp\n

Question

当我执行以下命令时，我收到取消拾取错误

rdd = sc.parallelize([('HOMICIDE', {'2017': 1}), 
('DECEPTIVE PRACTICE', {'2015': 2, '2017': 2, '2016': 8}), 
('ROBBERY', {'2016': 2})])

rdd.flatMapValues(dict.items).collect()

错误如下，在字典值上使用flatMapValues是否有问题

^{pr2}$

Answer 1

rdd = sc.parallelize([('HOMICIDE', {'2017': 1}), 
                      ('DECEPTIVE PRACTICE', {'2015': 2, '2017': 2, '2016': 8}), 
                      ('ROBBERY', {'2016': 2})])

rdd.flatMapValues(lambda data: data.items()).collect()

[('HOMICIDE', ('2017', 1)),
 ('DECEPTIVE PRACTICE', ('2015', 2)),
 ('DECEPTIVE PRACTICE', ('2017', 2)),
 ('DECEPTIVE PRACTICE', ('2016', 8)),
 ('ROBBERY', ('2016', 2))]

在dict.项目是方法描述符。您必须提供一个函数来通知flatmap如何解压缩这些值。我通过将labmda函数传递给flatMap函数来实现这一点。在