在python或pyspark中，如何删除不受顺序影响的列之间的重复值？

Question

我有一个数据帧，其中列之间有重复的值。我只想从dataframe中删除所有列的公共值

输入数据帧：

+----------+-----------+------------+
|    A     |    B      |   C        |
+----------+-----------+------------+
|  Food    |   Chill   |  Traveller |
+----------+-----------+------------+
|  Chill   |   Food    |  Chill     |
+----------+-----------+------------+
| Traveller|   Glamour |  Glamour   |
+----------+-----------+------------+
|  Novel   | Traveller |   Party    |
+----------+-----------+------------+
|  Pub     |   Party   |  Driving   |
+----------+-----------+------------+    

输出数据帧：

+----------+-----------+------------+
|    A     |    B      |   C        |
+----------+-----------+------------+
|  Food    |   Party   |  Driving   |
+----------+-----------+------------+
|  Novel   |   Food    |    Party   |
+----------+-----------+------------+
|   Pub    |   Glamour |  Glamour   |
+----------+-----------+------------+

在python或pyspark中有什么有效的方法可以做到这一点吗？你知道吗

Answer 1

可以使用set intersection和aggregate

>>> tr = set(df.A.values) & set(df.B.values) & set(df.C.values)
>>> df[~df.isin(tr)].agg(lambda s: s[~pd.isnull(s)].values)

    A       B       C
0   Food    Food    Glamour
1   Novel   Glamour Party
2   Pub     Party   Driving