如何从Dask中键维度不一致的字典列表中创建数据帧？

Question

对Dask来说有些陌生，但由于大多数操作都是惰性的，我如何才能使这样一个基本案例在规模上工作

import dask.dataframe as dd
import dask.bag as db

dataset = [
    dict(a = 1, b = 2, c = 3),
    dict(a = 3, b = 4, d = 5, e = 5),
    dict(a = 2, x = 1, y = 2, z = 3, q = 5)
    # etc...
]

dag_data = db.from_sequence(dataset)
dag_data.to_dataframe() 

在一个只有熊猫的世界里，我可以映射pd.Series，但问题是这些操作在计算之后才被计算。上面的代码生成一个数据帧，其中只包含来自第一条记录（“a”、“b”、“c”）的特征

执行结果：

| a | b | c | d | e | q | x | y | z |
---------------------------------------
| 1 | 2 | 3 | - | - | - | - | - | - |
| 3 | 4 | - | 5 | 5 | - | - | - | - |
| 2 | - | - | - | - | 5 | 1 | 2 | 3 | 

Answer 1

请把这当作一个延伸的评论，而不是一个答案

在熊猫中，只有你有

import pandas as pd

dataset = [
    dict(a = 1, b = 2, c = 3),
    dict(a = 3, b = 4, d = 5, e = 5),
    dict(a = 2, x = 1, y = 2, z = 3, q = 5)
]

df = pd.DataFrame(dataset)

而且{}是

   a    b    c    d    e    x    y    z    q
0  1  2.0  3.0  NaN  NaN  NaN  NaN  NaN  NaN
1  3  4.0  NaN  5.0  5.0  NaN  NaN  NaN  NaN
2  2  NaN  NaN  NaN  NaN  1.0  2.0  3.0  5.0

但是如果你搬到达斯克，我发现唯一可能的解决办法是

import pandas as pd
import dask.dataframe as dd
from dask import delayed, compute

dataset = [
    dict(a = 1, b = 2, c = 3),
    dict(a = 3, b = 4, d = 5, e = 5),
    dict(a = 2, x = 1, y = 2, z = 3, q = 5)
]

def fun(d):
    return pd.DataFrame(d, index=[0])

lst = [delayed(fun)(l) for l in dataset]

df = dd.concat(compute(lst)[0])

在我看来，这根本没有效率。看看是否有一种合适的方法来获得相同的输出是很有趣的