在数据库中的列组合中运行穷举搜索

Question

我正在一个数据库中对列的组合进行彻底的搜索。没有目标函数，因此没有优化。只是一系列详尽的df过滤器

我有一套标准化的文件。对于每一个，我构建一个df，其结构如下：

客户‘A’、‘B’、‘C’、…'K、Metric1、Metric2、Metric3

A-K列是我希望过滤df的特性。使用itertools，我从这些COL创建了5的所有独特组合

“Metric1”-“Metric3”列包含其他值，我想在过滤df后计算这些值的平均值

df有一个索引“Customer”

# Features
featureList =  ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K']
numInputs = 5

# Build a list of unique parameter combinations (list of tuples)
AllParms = []
AllParms = list(itertools.combinations(featureList, numInputs))

# Create a list of integers to identify the iteration, i1, i2, i3, etc.
iteration = list(range(1, len(AllParms) + 1))

# Loop thru files
for file in filenames:

    # Read data file.
    df = pd.read_csv(file, index_col='Customer', header=0)

    # Loop thru parameter sets
    for j in range(len(AllParms)):

        ''' 
        Get a unique parameter set (an element from the list of tuples).
        Parse tuple into variables to create df booleans 
        Get parms from 'AllParms' and iteration number from 'iteration'
        '''         

        parmToIterate = AllParms[j]

            parmn = 'i' + str(iteration[j])
            parmA = parmToIterate[0]
            parmB = parmToIterate[1]
            parmC = parmToIterate[2]
            parmD = parmToIterate[3]
            parmE = parmToIterate[4]
            concatStr = parmA + '_' + parmB + '_' + parmC + '_' + parmD + '_' + parmE


            ''' Filter df '''
            # Method 1
            df[parmn] = (
            (df[parmA] > 0) &
            (df[parmB] > 0) &
            (df[parmC] > 0) &
            (df[parmD] > 0) &
            (df[parmE] > 0)).astype(str)
            df2 = df.loc[df[parmn].isin(['True'])]

            # Method 2
            Cond1 = df[parmA] > 0
            Cond2 = df[parmB] > 0
            Cond3 = df[parmC] > 0
            Cond4 = df[parmD] > 0
            Cond5 = df[parmE] > 0
            AllCond = Cond1 & Cond2 & Cond3 & Cond4 & Cond5
            df2 = df[AllCond]


            ''' Calc Metrics for Filtered Rows'''
            Metric1_mean = round(df2['Metric1'].mean(),3)
            Metric2_mean = round(df2['Metric2'].mean(),3)
            Metric3_mean = round(df2['Metric3'].mean(),3)

            ''' Join metrics for all parm sets and unique parm string '''

问题:

1. 上面的代码工作得很好，但我读过很多关于动态创建df列的负面评论。创建df[element from a list或tuple]=something有什么不对？当循环通过COL集合时，有什么替代方案

2. 方法2比方法1快6.7倍。我知道方法1是纯行操作，但方法2不是吗

Answer 1

1。您可以尝试：

conditions = (df[list_of_columns]>0).all(1)

1. 问题是方法1将布尔值转换为字符串，并对字符串进行比较，这通常比较慢