从元组列表中删除重复项

Question

我有一个元组列表，不幸的是其中包含重复的元组，如下所示：

[(67, u'top-coldestcitiesinamerica'), (66, u'ecofriendlyideastocelebrateindependenceday-phpapp'), (65, u'a-b-c-ca-d-ab-ea-d-c-c'), (64, u'a-b-c-ca-d-ab-ea-d-c-c'), (63, u'alexandre-meybeck-faowhatisclimate-smartagriculture-backgroundopportunitiesandchallenges'), (62, u'ghgemissions'), (61, u'top-coldestcitiesinamerica'), (58, u'infographicthe-stateofdigitaltransformationaltimetergroup'), (57, u'culture'), (55, u'cas-k-ihaveanidea'), (54, u'trendsfor'), (53, u'batteryimpedance'), (52, u'evs-howey-full'), (51, u'bericht'), (49, u'classiccarinsurance'), (47, u'uploaded_file'), (46, u'x_file'), (45, u's-s-main'), (44, u'vehicle-propulsion'), (43, u'x_file')]

问题是元组的第一个元素（基于0的排序）是我要检查重复项的条目。所以，我可以看到：

(67, u'top-coldestcitiesinamerica')
(61, u'top-coldestcitiesinamerica')

..是重复的，我想删除其中一个（类似于set）。因此，在最后，我希望有一个没有重复的元组的干净列表（即元组的第一个元素没有重复）：

[(67, u'top-coldestcitiesinamerica'), (66, u'ecofriendlyideastocelebrateindependenceday-phpapp'), (65, u'a-b-c-ca-d-ab-ea-d-c-c') (63, u'alexandre-meybeck-faowhatisclimate-smartagriculture-backgroundopportunitiesandchallenges'), (62, u'ghgemissions'), (58, u'infographicthe-stateofdigitaltransformationaltimetergroup'), (57, u'culture'), (55, u'cas-k-ihaveanidea'), (54, u'trendsfor'), (53, u'batteryimpedance'), (52, u'evs-howey-full'), (51, u'bericht'), (49, u'classiccarinsurance'), (47, u'uploaded_file'), (46, u'x_file'), (45, u's-s-main'), (44, u'vehicle-propulsion')]

我怎样才能以一种Python的方式达到这个目的呢？ 谢谢！你知道吗

Answer 1

作为一个替代答案，您可以使用itertools.groupby()，如果您有一个巨大的列表，这可能会很有帮助，但不如set：

>>> from itertools import groupby
>>> from operator import itemgetter
>>> [next(g) for _,g in groupby(sorted(l,key=itemgetter(1)),itemgetter(1))]
[(65, u'a-b-c-ca-d-ab-ea-d-c-c'), (63, u'alexandre-meybeck-faowhatisclimate-smartagriculture-backgroundopportunitiesandchallenges'), (53, u'batteryimpedance'), (51, u'bericht'), (55, u'cas-k-ihaveanidea'), (49, u'classiccarinsurance'), (57, u'culture'), (66, u'ecofriendlyideastocelebrateindependenceday-phpapp'), (52, u'evs-howey-full'), (62, u'ghgemissions'), (58, u'infographicthe-stateofdigitaltransformationaltimetergroup'), (45, u's-s-main'), (67, u'top-coldestcitiesinamerica'), (54, u'trendsfor'), (47, u'uploaded_file'), (44, u'vehicle-propulsion'), (46, u'x_file')]

Answer 2

我做这件事非常简单和容易。你知道吗

lst=[(67, u'top-coldestcitiesinamerica'), (66, u'ecofriendlyideastocelebrateindependenceday-phpapp'), (65, u'a-b-c-ca-d-ab-ea-d-c-c'), (64, u'a-b-c-ca-d-ab-ea-d-c-c'), (63, u'alexandre-meybeck-faowhatisclimate-smartagriculture-backgroundopportunitiesandchallenges'), (62, u'ghgemissions'), (61, u'top-coldestcitiesinamerica'), (58, u'infographicthe-stateofdigitaltransformationaltimetergroup'), (57, u'culture'), (55, u'cas-k-ihaveanidea'), (54, u'trendsfor'), (53, u'batteryimpedance'), (52, u'evs-howey-full'), (51, u'bericht'), (49, u'classiccarinsurance'), (47, u'uploaded_file'), (46, u'x_file'), (45, u's-s-main'), (44, u'vehicle-propulsion'), (43, u'x_file')]

lst2 = [] #empty list to fill with unique tuples
lst_banned = [] #empty list to fill with banned elements

for tup in lst:
    if tup[-1] not in lst_banned:
        lst_banned.append(tup[-1])
        lst2.append(tup)

lst=lst2
del lst2
del lst_banned

Answer 3

您可以使用来自How do you remove duplicates from a list in whilst preserving order?的set方法，使用x[1]作为唯一标识符：

def unique_second_element(seq):
    seen = set()
    seen_add = seen.add
    return [x for x in seq if not (x[1] in seen or seen_add(x[1]))]

请注意，如果您想保留最后一次出现的情况，也可以使用OrderedDict方法；对于第一次出现的情况，您必须反转输入，然后再次反转输出。你知道吗

您可以通过支持key函数使其更通用：

def unique_preserve_order(seq, key=None):
    if key is None:
        key = lambda elem: elem
    seen = set()
    seen_add = seen.add
    augmented = ((key(x), x) for x in seq)
    return [x for k, x in augmented if not (k in seen or seen_add(k))]

然后使用

import operator

unique_preserve_order(yourlist, key=operator.itemgetter(1))

演示：

>>> def unique_preserve_order(seq, key=None):
...     if key is None:
...         key = lambda elem: elem
...     seen = set()
...     seen_add = seen.add
...     augmented = ((key(x), x) for x in seq)
...     return [x for k, x in augmented if not (k in seen or seen_add(k))]
... 
>>> from pprint import pprint
>>> import operator
>>> yourlist = [(67, u'top-coldestcitiesinamerica'), (66, u'ecofriendlyideastocelebrateindependenceday-phpapp'), (65, u'a-b-c-ca-d-ab-ea-d-c-c'), (64, u'a-b-c-ca-d-ab-ea-d-c-c'), (63, u'alexandre-meybeck-faowhatisclimate-smartagriculture-backgroundopportunitiesandchallenges'), (62, u'ghgemissions'), (61, u'top-coldestcitiesinamerica'), (58, u'infographicthe-stateofdigitaltransformationaltimetergroup'), (57, u'culture'), (55, u'cas-k-ihaveanidea'), (54, u'trendsfor'), (53, u'batteryimpedance'), (52, u'evs-howey-full'), (51, u'bericht'), (49, u'classiccarinsurance'), (47, u'uploaded_file'), (46, u'x_file'), (45, u's-s-main'), (44, u'vehicle-propulsion'), (43, u'x_file')]
>>> pprint(unique_preserve_order(yourlist, operator.itemgetter(1)))
[(67, u'top-coldestcitiesinamerica'),
 (66, u'ecofriendlyideastocelebrateindependenceday-phpapp'),
 (65, u'a-b-c-ca-d-ab-ea-d-c-c'),
 (63,
  u'alexandre-meybeck-faowhatisclimate-smartagriculture-backgroundopportunitiesandchallenges'),
 (62, u'ghgemissions'),
 (58, u'infographicthe-stateofdigitaltransformationaltimetergroup'),
 (57, u'culture'),
 (55, u'cas-k-ihaveanidea'),
 (54, u'trendsfor'),
 (53, u'batteryimpedance'),
 (52, u'evs-howey-full'),
 (51, u'bericht'),
 (49, u'classiccarinsurance'),
 (47, u'uploaded_file'),
 (46, u'x_file'),
 (45, u's-s-main'),
 (44, u'vehicle-propulsion')]