我试图将两个列表中的公司名称进行匹配,以检查列表a中的某个公司是否确实列在B列表中。由于a公司的名称以各种不同的形式书写,我倾向于使用余弦相似性进行匹配。 为此,我关注了Ran Tavory在博客上的留言:datasets-of-company-1be4b1b2871e" rel="nofollow noreferrer">Link Here
以下是概要:
- Calculate TF-IDF matrices on the driver.
- Parallelize matrix A; Broadcast matrix B
- Each worker now flatMaps its chunk of work by multiplying its chunk of matrix A with the entire matrix B. So if a worker operates on A[0:99] then it would multiply these hundred rows and return the result of, say A[13] matches a name found in B[21]. Multiplication is done using numpy.
- The driver would collect back all the results from the different workers and match the indices (A[13] and B[21]) to the actual names in the original dataset — and we’re done!
我可以运行注释中描述的代码,但其中有一部分似乎有点奇怪:
b_mat_dist = broadcast_matrix(a_mat)
当广播一个_mat以及并行化一个_mat时,我会得到一个逻辑结果,即每个公司的名称都完全匹配(因为我们在同一个源中查找)。在
当我尝试广播b_mat:b_mat_dist=broadcast_matrix(b_mat)时,我得到以下错误:Incompatible dimension for X and Y matrices: X.shape[1] == 56710 while Y.shape[1] == 2418
任何帮助将不胜感激! 提前谢谢!在
这是我的代码:
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
from pyspark.sql import SQLContext, SparkSession
from pyspark import SparkContext
from scipy.sparse import csr_matrix
vectorizer = TfidfVectorizer()
if 'sc' in locals():
sc.stop()
sc = SparkContext("local", "Simple App")
pd.set_option('display.max_colwidth', -1)
RefB = pd.read_excel('Ref.xlsx')
ToMatchB = pd.read_excel('ToMatch.xlsx')
Ref = RefB['CLT_company_name']
ToMatch = ToMatchB ['Name1']
a_mat = vectorizer.fit_transform(Ref)
b_mat = vectorizer.fit_transform(ToMatch)
def find_matches_in_submatrix(sources, targets, inputs_start_index,
threshold=.8):
cosimilarities = cosine_similarity(sources, targets)
for i, cosimilarity in enumerate(cosimilarities):
cosimilarity = cosimilarity.flatten()
# Find the best match by using argsort()[-1]
target_index = cosimilarity.argsort()[-1]
source_index = inputs_start_index + i
similarity = cosimilarity[target_index]
if cosimilarity[target_index] >= threshold:
yield (source_index, target_index, similarity)
def broadcast_matrix(mat):
bcast = sc.broadcast((mat.data, mat.indices, mat.indptr))
(data, indices, indptr) = bcast.value
bcast_mat = csr_matrix((data, indices, indptr), shape=mat.shape)
return bcast_mat
def parallelize_matrix(scipy_mat, rows_per_chunk=100):
[rows, cols] = scipy_mat.shape
i = 0
submatrices = []
while i < rows:
current_chunk_size = min(rows_per_chunk, rows - i)
submat = scipy_mat[i:i + current_chunk_size]
submatrices.append((i, (submat.data, submat.indices,
submat.indptr),
(current_chunk_size, cols)))
i += current_chunk_size
return sc.parallelize(submatrices)
a_mat_para = parallelize_matrix(a_mat, rows_per_chunk=100)
b_mat_dist = broadcast_matrix(b_mat)
results = a_mat_para.flatMap(
lambda submatrix:
find_matches_in_submatrix(csr_matrix(submatrix[1],
shape=submatrix[2]),
b_mat_dist,
submatrix[0]))
尝试为两个TfidVectorizer对象均衡词汇表:
同样基于您的目标:
^{pr2}$对我来说是个更好的选择。在
相关问题 更多 >
编程相关推荐