Group by fuzzy和groupby的模糊字符串匹配

Question

我有一个随机单词和名字的数据集，我正在尝试对所有相似的单词和名字进行分组。因此，鉴于以下数据框架：

     Name           ID            Value
0    James           1             10
1    James 2         2             142
2    Bike            3             1
3    Bicycle         4             1197
4    James Marsh     5             12
5    Ants            6             54
6    Job             7             6
7    Michael         8             80007  
8    Arm             9             47 
9    Mike K          10            9
10   Michael k       11            1

我的伪代码类似于：

import pandas as pd
from fuzzywuzzy import fuzz

minratio = 95
for idx1, name1 in df['Name'].iteritems():
   for idx2, name2 in df['Name'].iteritems():
      ratio = fuzz.WRatio(name1, name2)
      if ratio > minratio:
          grouped = df.groupby(['Name', 'ID'])['Value']\
                        .agg(Total_Value='sum', Group_Size='count')

这将为我提供所需的输出：

print(grouped)
     Name           ID            Total_Value          Group_Size
0    James           1             164                     3 # All James' grouped
2    Bike            3             1198                    2 # Bike's and Bicycles grouped
5    Ants            6             54                      1 
6    Job             7             6                       1
7    Michael         8             80017                   3 # Mike's and Michael's grouped
8    Arm             9             47                      1

显然，这是行不通的，老实说，我甚至不确定这是否可行，但这就是我试图实现的目标。任何能让我走上正轨的建议都是有用的

Answer 1

使用affinity propagation clustering（不完美，但可能是一个起点）：

import pandas as pd
import numpy as np
import io
from fuzzywuzzy import fuzz
from scipy import spatial
import sklearn.cluster

s="""Name           ID            Value
0    James           1             10
1    James 2         2             142
2    Bike            3             1
3    Bicycle         4             1197
4    James Marsh     5             12
5    Ants            6             54
6    Job             7             6
7    Michael         8             80007  
8    Arm             9             47 
9    Mike K          10            9
10   Michael k       11            1"""
df = pd.read_csv(io.StringIO(s),sep='\s\s+',engine='python')

names = df.Name.values
sim = spatial.distance.pdist(names.reshape((-1,1)), lambda x,y: fuzz.WRatio(x,y))
affprop = sklearn.cluster.AffinityPropagation(affinity="precomputed", random_state=None)
affprop.fit(spatial.distance.squareform(sim))

res = df.groupby(affprop.labels_).agg(
        Names=('Name',','.join),
        First_ID=('ID','first'),
        Total_Value=('Value','sum'),
        Group_Size=('Value','count')
        )

结果

                                Names  First_ID  Total_Value  Group_Size
0  James,James 2,James Marsh,Ants,Arm         1          265           5
1                        Bike,Bicycle         3         1198           2
2                                 Job         7            6           1
3            Michael,Mike K,Michael k         8        80017           3