用于大规模杀伤性武器相似性的循环列

Question

我有两个数据帧。两者都有两列。我想使用wmd为列source_label中的每个实体找到与列target_label中的实体最接近的匹配项。但是，在最后，我希望有一个数据框，其中包含所有4列与实体相关的数据

df1

,source_Label,source_uri
'neuronal ceroid lipofuscinosis 8',"http://purl.obolibrary.org/obo/DOID_0110723"
'autosomal dominant distal hereditary motor neuronopathy',"http://purl.obolibrary.org/obo/DOID_0111198"

df2

,target_label,target_uri
'neuronal ceroid ',"http://purl.obolibrary.org/obo/DOID_0110748"
'autosomal dominanthereditary',"http://purl.obolibrary.org/obo/DOID_0111110"

预期结果

,source_label, target_label, source_uri, target_uri, wmd score
'neuronal ceroid lipofuscinosis 8', 'neuronal ceroid ', "http://purl.obolibrary.org/obo/DOID_0110723", "http://purl.obolibrary.org/obo/DOID_0110748", 0.98
'autosomal dominant distal hereditary motor neuronopathy', 'autosomal dominanthereditary', "http://purl.obolibrary.org/obo/DOID_0111198", "http://purl.obolibrary.org/obo/DOID_0111110", 0.65

dataframe太大了，我正在寻找一些更快的方法来迭代这两个标签列。到目前为止，我试过：

list_distances = []
temp = []

def preprocess(sentence):
    return [w for w in sentence.lower().split()]

entity = df1['source_label']
target = df2['target_label']

 for i in tqdm(entity):
    for j in target:
        wmd_distance = model.wmdistance(preprocess(i), preprocess(j))
        temp.append(wmd_distance)
    list_distances.append(min(temp))
# print("list_distances", list_distances)
WMD_Dataframe = pd.DataFrame({'source_label': pd.Series(entity),
                              'target_label': pd.Series(target),
                              'source_uri': df1['source_uri'],
                              'target_uri': df2['target_uri'],
                              'wmd_Score': pd.Series(list_distances)}).sort_values(by=['wmd_Score'])
WMD_Dataframe = WMD_Dataframe.reset_index()

首先，这段代码工作得不好，因为其他两列直接来自dfs，没有考虑实体与uri的关系。 当实体以百万计时，如何使其更快。提前谢谢

Answer 1

快速修复：

closest_neighbour_index_df2 = []


def preprocess(sentence):
    return [w for w in sentence.lower().split()]



 
for i in tqdm(entity):
    temp = []
    for j in target:
        wmd_distance = model.wmdistance(preprocess(i), preprocess(j))
        temp.append(wmd_distance)
    # maybe assert to make sure its always right
    closest_neighbour_index_df2.append(np.argmin(np.array(temp))) 
    # return argmin to return index rather than the value. 
    
# Add the indices from df2 to df1

df1['closest_neighbour'] = closest_neighbour_index_df2 
# add information to respective row from df2 using the closest_neighbour column