基于datafram的优化迭代

Question

我有这样的数据帧：

            timestamp    battery_state  battery_level
0   2017-10-08 13:42:02  Charging       0.94
1   2017-10-08 13:45:43  Charging       0.95
2   2017-10-08 13:49:08  Charging       0.96
3   2017-10-08 13:54:07  Charging       0.97
4   2017-10-08 13:57:26  Charging       0.98
5   2017-10-08 14:01:35  Charging       0.99
6   2017-10-08 14:03:03  Full           1.00
7   2017-10-08 14:17:19  Charging       0.98
8   2017-10-08 14:26:05  Charging       0.97
9   2017-10-08 14:46:10  Charging       0.98
10  2017-10-08 14:47:47  Full           1.00
11  2017-10-08 16:36:24  Charging       0.91
12  2017-10-08 16:40:32  Charging       0.92
13  2017-10-08 16:47:58  Charging       0.93
14  2017-10-08 16:51:51  Charging       0.94
15  2017-10-08 16:55:26  Charging       0.95

正如你们在这个数据帧中看到的，我有3个与设备充电周期相对应的样本子集：

• 样本0到6
• 样品7至10
• 样品11至15

注意：充电周期并不总是完全处于样品11到15的状态

我们的目标是将这3个周期转换成一个变量，并在它们建立时进行处理。你知道吗

为此，我编写了以下代码：

previous_index = 0 #stores the initial index of each period

for index in islice(device_charge_samples.index, 1, None): #use islice because the first row does not have privious sample to compare

    #creates a period by comparing each line two by two
    if device_charge_samples.get_value(index, 'battery_level') < device_charge_samples.get_value(index - 1, 'battery_level'):
         subset = device_charge_samples[previous_index:index].reset_index(drop=True)

         #Process subset function here

         previous_index = index

    #last period case
    if index == len(device_charge_samples) - 1:
         subset = device_charge_samples[previous_index:index + 1].reset_index(drop=True)

         #Process subset function here

我已经更换了设备\u充电_示例.iteraterows（）用于设备充电_样本.索引打开for循环，我更换设备\u充电_示例.loc[索引，'电池电量]用于设备充电_samples.get\u值（索引，‘电池电量’），两者都有很大帮助。你知道吗

我还可以做其他优化吗？，比如使用dataframe apply函数（它似乎可以作为每行的for循环，但我不知道在这种情况下如何使用它，或者是否值得使用它），或者我可以在解决方案中使用的任何其他优化

Answer 1

您可以使用np.split()where battery_state == 'Full'并删除这些行。你知道吗

m = df['battery_state'] == 'Full'
for subset in np.split(df[~m],df.index[m] - np.arange(sum(m))):
    #1000 loops, best of 3: 783 µs per loop
    # do something with subset here

或者像DJK用cumsum所说的那样（但这里是一个更紧凑的版本，以便于公平计时）

m  = df.battery_state == 'Full'
for idx, subset in df[~m].groupby(m.cumsum()):
    # 1000 loops, best of 3: 999 µs per loop
    # do something with subset here

完整示例：

import pandas as pd
import numpy as np

data = '''\
timestamp            battery_state  battery_level
2017-10-08T13:42:02  Charging       0.94
2017-10-08T13:45:43  Charging       0.95
2017-10-08T13:49:08  Charging       0.96
2017-10-08T13:54:07  Charging       0.97
2017-10-08T13:57:26  Charging       0.98
2017-10-08T14:01:35  Charging       0.99
2017-10-08T14:03:03  Full           1.00
2017-10-08T14:17:19  Charging       0.98
2017-10-08T14:26:05  Charging       0.97
2017-10-08T14:46:10  Charging       0.98
2017-10-08T14:47:47  Full           1.00
2017-10-08T16:36:24  Charging       0.91
2017-10-08T16:40:32  Charging       0.92
2017-10-08T16:47:58  Charging       0.93
2017-10-08T16:51:51  Charging       0.94
2017-10-08T16:55:26  Charging       0.95'''

df = pd.read_csv(pd.compat.StringIO(data), sep='\s+', parse_dates=['timestamp'])

m = df['battery_state'] == 'Full'
for subset in np.split(df[~m],df.index[m] - np.arange(sum(m))):
    print(subset)

退货：

            timestamp battery_state  battery_level
0 2017-10-08 13:42:02      Charging           0.94
1 2017-10-08 13:45:43      Charging           0.95
2 2017-10-08 13:49:08      Charging           0.96
3 2017-10-08 13:54:07      Charging           0.97
4 2017-10-08 13:57:26      Charging           0.98
5 2017-10-08 14:01:35      Charging           0.99
            timestamp battery_state  battery_level
7 2017-10-08 14:17:19      Charging           0.98
8 2017-10-08 14:26:05      Charging           0.97
9 2017-10-08 14:46:10      Charging           0.98
             timestamp battery_state  battery_level
11 2017-10-08 16:36:24      Charging           0.91
12 2017-10-08 16:40:32      Charging           0.92
13 2017-10-08 16:47:58      Charging           0.93
14 2017-10-08 16:51:51      Charging           0.94
15 2017-10-08 16:55:26      Charging           0.95

Answer 2

首先创建一个列，使用cumsum对数据帧进行唯一的分段

df['group'] = (df.battery_state == 'Full').cumsum().shift(1).fillna(0)

现在您可以迭代组，而不是迭代行

for index, frame in df.groupby('group'):
    subsetFunction(frame)