pyspark:Get columns 根据其他记录

Question

我有一个像这样的数据帧

membershipAccountNbr            cntryRetailChannelCustId
111590058               1010015900581000010101
214100897               1010041008972100010101
104100897               1010041008971000010101

另一个是这样的：

membershipAccountNbr    parentMembershipNbr
111590058                   111590058
214100897                   104100897

我的目标是让它看起来像：

membershipAccountNbr parentMembershipNbr parentCustId
111590058               111590058    1010015900581000010101
214100897               104100897    1010041008971000010101

我试过使用连接，但它们给出了一个模糊错误。我是Pyspark的新手，所以请帮帮我

Answer 1

假设df1是

+          +            +
|membershipAccountNbr|cntryRetailChannelCustId|
+          +            +
|           111590058|    10100159005810000...|
|           214100897|    10100410089721000...|
|           104100897|    10100410089710000...|
+          +            +

和df2

+          +         -+
|membershipAccountNbr|parentMembershipNbr|
+          +         -+
|           111590058|          111590058|
|           214100897|          104100897|
+          +         -+

然后你就跑

df1.join(df2, on="membershipAccountNbr", how="right").select(
    col("membershipAccountNbr"),
    col("parentMembershipNbr"),
    col("cntryRetailChannelCustId").alias("parentCustId"),
).show()

结果是这样的

+          +         -+          +
|membershipAccountNbr|parentMembershipNbr|        parentCustId|
+          +         -+          +
|           111590058|          111590058|10100159005810000...|
|           214100897|          104100897|10100410089721000...|
+          +         -+          +