如何编写在Python中的20个不同csv文件上运行该函数的函数？

Question

我有一个19个csv文件的目录，每个文件包含一个学生注册号和他们的名字的列表。有两个单独的文件名为quiz1和quiz2，这两个文件都包含所有参加这些测验的学生的信息，以及他们的姓名和获得的总分。在每一个测验中获得的分数必须被分成不同的栏，以及一个“noofpresent”栏，显示他们参加那个特定测验的情况。你知道吗

我的任务是解析所有这些文件并创建一个基本上如下面所示的数据帧。上图显示了总共19个批中的5个批。你知道吗

虽然我已经填写了Batch4的相关字段，如图所示，但我意识到对18个文件重复这个过程是疯狂的。你知道吗

如何编写一个程序或函数来完成剩余18个批次的所有操作？我只需要一个如何继续与其余18个文件的自动化逻辑的想法。你知道吗

第9批的ex（例如）：

这是我需要为19个批中的每个批复制的代码：

import pandas as pd

spath = 'd:\\a2\\studentlist.csv'
q1path = 'd:\\a2\\quiz\\quiz1.csv'
q2path = 'd:\\a2\\quiz\\quiz2.csv'
b1path = 'd:\\a2\\batchwiselist\\1.csv'
b9path = 'd:\\a2\\batchwiselist\\9.csv'
tpath = 'd:\\a2\\testcasestudent.txt'

# the final dataframe that needs to be created and filled up eventually
idx = pd.MultiIndex.from_product([['batch1', 'batch2', 'batch3', 'batch4', 'batch9'], ['quiz1', 'quiz2']])
cols=['noofpresent', 'lesserthan50', 'between50and60', 'between60and70', 'between70and80', 'greaterthan80']
statdf = pd.DataFrame('-', idx, cols)


# ============BATCH 9===================]

# ----------- QUIZ 1 -----------]

# Master list of students in Batch 9
b9 = pd.read_csv(b9path, usecols=['studentName', 'admissionNumber'])
b9.rename(columns={'studentName' : 'Firstname'}, inplace=True)
# To match column from quiz1.csv to batch9.csv to for merger

# Master list of all who attended Quiz1
q1 = pd.read_csv(q1path, usecols = ['Firstname', 'Grade/10.00', 'State'], na_values = ['-', 'In progress', np.NaN])
q1.dropna(inplace=True)
q1['Grade/10.00'] = q1['Grade/10.00'] * 10
# Multiplying the grades by 10 to mark against 100 instead of 10

# Merge batch9 list of names to list of quiz1 on their firstname column
q1b9 = pd.merge(b9, q1)
q1b9 = q1.loc[q1['Firstname'].isin(b9.Firstname)]        # checking if the name exits in either lists
q1b9.reset_index(inplace=True)
#print(q1b9)

lt50 = q1b9.loc[(q1b9['Grade/10.00'] < 50)]         
#findout list of students whose grades are lesser than 50
out9q1 = (lt50['Grade/10.00'].count())
# print(out9q1) to just get the count of number of students who got <50 quiz1 from batch9

# Similar process for quiz2 below for batch9.
# -------------------- QUIZ 2 ------------------]

# Master list of all who attended Quiz2
q2 = pd.read_csv(q2path, usecols = ['Firstname', 'Grade/10.00', 'State'], na_values = ['-', 'In progress', np.NaN])
q2.dropna(inplace=True)
q2['Grade/10.00'] = q2['Grade/10.00'] * 10

# Merge B1 to Q2
q2b9 = pd.merge(b9, q2)
q2b9 = q2.loc[q2['Firstname'].isin(b9.Firstname)]
q2b9.reset_index(inplace=True)


q2b9.loc[(q2b9['Grade/10.00'] <= 50)].count()
lt50 = q2b9.loc[(q2b9['Grade/10.00'] < 50)]
out9q2 = (lt50['Grade/10.00'].count())
# print(out9q2)

上面的代码计算的是在任何一次测验中成绩低于50分的所有学生。我对第4批也做了类似的工作。我需要复制它，这样一个函数就可以对所有剩余的（17-18）批处理执行此操作。你知道吗

Answer 1

在下面的代码中，我已经生成了所有csv路径并逐个加载，然后执行所有过程，然后生成的数据帧保存在数据帧列表中，如[[batch1\u q1\u result，batch1\u q2\u result]，[batch2\u q1\u result，batch2\u q2\u result]…]

def doAll(baseBatchPath, numberOfBatches):
    batchResultListAll = [] # this will store the resulted dataframes
    spath = 'd:\\a2\\studentlist.csv'
    q1path = 'd:\\a2\\quiz\\quiz1.csv'
    q2path = 'd:\\a2\\quiz\\quiz2.csv'
    tpath = 'd:\\a2\\testcasestudent.txt'
    # the final dataframe that needs to be created and filled up eventually
    idx = pd.MultiIndex.from_product([['batch1', 'batch2', 'batch3', 'batch4', 'batch9'], ['quiz1', 'quiz2']])
    cols=['noofpresent', 'lesserthan50', 'between50and60', 'between60and70', 'between70and80', 'greaterthan80']
    statdf = pd.DataFrame('-', idx, cols)

    # Master list of all who attended Quiz1
    q1 = pd.read_csv(q1path, usecols = ['Firstname', 'Grade/10.00', 'State'], na_values = ['-', 'In progress', np.NaN])
    q1.dropna(inplace=True)
    q1['Grade/10.00'] = q1['Grade/10.00'] * 10
    # Master list of all who attended Quiz2
    q2 = pd.read_csv(q2path, usecols = ['Firstname', 'Grade/10.00', 'State'], na_values = ['-', 'In progress', np.NaN])
    q2.dropna(inplace=True)
    q2['Grade/10.00'] = q2['Grade/10.00'] * 10

    # generate each batch file path and do other works
    for batchId in range(numberOfBatches-1):
        batchCsvPath = baseBatchPath + str(batchId+1) + ".csv"
        # Master list of students in Batch 9
        batch = pd.read_csv(batchCsvPath, usecols=['studentName', 'admissionNumber'])
        batch.rename(columns={'studentName' : 'Firstname'}, inplace=True)
        # Merge eachBatch list of names to list of quiz1 on their firstname column
        q1batch = pd.merge(batch, q1)
        q1batch = q1.loc[q1['Firstname'].isin(batch.Firstname)]        # checking if the name exits in either lists
        q1batch.reset_index(inplace=True)
        #print(q1batch)

        lt50 = q1batch.loc[(q1batch['Grade/10.00'] < 50)]         
        #findout list of students whose grades are lesser than 50
        outBatchq1 = (lt50['Grade/10.00'].count())
        # print(outBatchq1) to just get the count of number of students who got <50 quiz1 from batch -> batchId

        #do same for quiz 2

        # Merge each Batch to Q2
        q2batch = pd.merge(batch, q2)
        q2batch = q2.loc[q2['Firstname'].isin(batch.Firstname)]
        q2batch.reset_index(inplace=True)


        q2batch.loc[(q2batch['Grade/10.00'] <= 50)].count()
        lt50 = q2batch.loc[(q2batch['Grade/10.00'] < 50)]
        outBatchq2 = (lt50['Grade/10.00'].count())
        # print(outBatchq2)
        # finally save the resulted DF for later use
        batchResultListAll.append([q1batch, q2batch])


#call the function using base path and number of batch csv files        
doAll("d:\\\\a2\\\\batchwiselist\\\\", 18)

Answer 2

制作一个包含所有CSV文件路径的list对象，然后使用for循环来解析所有这些。显然，您将不得不在csv文件中使用动态file硬编码的地方调整代码 像这样：

csv_files = ['file1.csv','file2.csv2']
for file in csv_files:
      (YOUR CODE GOES HERE)