使用python应用程序进行多处理，以缩短当前36小时的运行时间

Question

我目前正在从事一个数据挖掘项目，该项目正在创建一个18000x18000的相似性矩阵

下面是构建矩阵的两种方法

def CreateSimilarityMatrix(dbSubsetData, distancePairsList):

    global matrix
    matrix = [ [0.0 for y in range(dbSubsetData.shape[0])] for x in range(dbSubsetData.shape[0])]
    for i in range(len(dbSubsetData)): #record1

        SimilarityArray = []
        start = time.time()
        for j in range(i+1, len(dbSubsetData)): #record2
            Similarity = GetDistanceBetweenTwoRecords(dbSubsetData, i, j, distancePairsList)

            #The similarities are all very small numbers which might be why the preference value needs to be so precise. 
            #Let's multiply the value by a scalar 10 to give the values more range. 
            matrix[i][j] = Similarity * 10.0
            matrix[j][i] = Similarity * 10.0

        end = time.time()
    return matrix

def GetDistanceBetweenTwoRecords(dbSubsetData, i, j, distancePairsList):

    Record1 = dbSubsetData.iloc[i]
    Record2 = dbSubsetData.iloc[j]
    columns = dbSubsetData.columns
    distancer    = 0.0
    distancec    = 0.0

    for i in range(len(Record1)):

        columnName = columns[i]
        Record1Value = Record1[i]
        Record2Value = Record2[i]

        if(Record1Value != Record2Value):
            ob = distancePairsList[distancePairsDict[columnName]-1]

            if(ob.attributeType == "String"):
                strValue = Record1Value+":"+Record2Value
                strValue2 = Record2Value+":"+Record1Value

                if strValue in ob.distancePairs:
                    val = ((ob.distancePairs[strValue])**2) 
                    val = val * -1
                    distancec = distancec + val
                elif strValue2 in ob.distancePairs:
                    val = ((ob.distancePairs[strValue2])**2)
                    val = val * -1
                    distancec = distancec + val

            elif(ob.attributeType == "Number"):
                val = ((Record1Value - Record2Value)*ob.getSignificance())**2 
                val = val * -1
                distancer = distancer + val

    distance = distancer + distancec
    return distance

每次迭代循环18000x19次（每行18000次，每个属性19次）。迭代的总数是（18000x18000x19）/2，因为它是对称的，因此我只需要做矩阵的一半。这将需要大约36个小时来完成，这是一个时间框架，我显然想剃下来

我认为多重处理是关键。由于每一行都独立地生成数字并将它们拟合到矩阵中，因此我可以使用CreateSimilarityMatrix运行多进程。所以我在函数中创建了这个，它将创建我的进程

matrix = [ [0.0 for y in range(SubsetDBNormalizedAttributes.shape[0])] for x in range(SubsetDBNormalizedAttributes.shape[0])] 
if __name__ == '__main__':    
    procs = []
    for i in range(4):
        proc = Process(target=CreateSimilarityMatrix, args=(SubsetDBNormalizedAttributes, distancePairsList, i, 4))
        procs.append(proc)
        proc.start()
        proc.join()

CreateSimilarityMatrix现在更改为

def CreateSimilarityMatrix(dbSubsetData, distancePairsList, counter=0, iteration=1):

    global Matrix

    for i in range(counter, len(dbSubsetData), iteration): #record1

        SimilarityArray = []
        start = time.time()
        for j in range(i+1, len(dbSubsetData)): #record2
            Similarity = GetDistanceBetweenTwoRecords(dbSubsetData, i, j, distancePairsList)
            #print("Similarity Between Records",i,":",j," is ", Similarity)

            #The similarities are all very small numbers which might be why the preference value needs to be so precise. 
            #Let's multiply the value by a scalar 10 to give the values more range. 
            Matrix[i][j] = Similarity * 10.0
            Matrix[j][i] = Similarity * 10.0

        end = time.time()
        print("Iteration",i,"took",end-start,"(s)")

现在是s-l-o-w。真的很慢。启动一个进程需要几分钟，然后启动下一个进程需要几分钟。我以为它们应该同时运行？我对这个过程的应用是否不正确

Answer 1

如果您使用的是CPython，那么就有一种称为全局解释器锁（global interpreter lock，GIL）的方法，这种方法在提高速度的同时很难实现多线程，反而会大大降低速度

如果要处理矩阵，请使用numpy，它肯定比常规Python快得多