基准测试和加速这个Python代码

Question

有人知道为什么test1b（）比test1a（）快得多吗？如何确定哪条线路是瓶颈，并选择替代功能来加快速度？请分享你的经验

import numpy as np
import pandas as pd
import time

def test1a():
    cols = 13
    rows = 10000000
    raw_data = np.random.randint(2, size=cols * rows).reshape(rows, cols)
    col_names = ['v01', 'v02', 'v03', 'v04', 'v05', 'v06', 'v07',
                 'v08', 'v09', 'v10', 'v11', 'v12', 'outcome']
    df = pd.DataFrame(raw_data, columns=col_names)
    df['v11'] = df['v03'].apply(lambda x: ['t1', 't2', 't3', 't4'][np.random.randint(4)])
    df['v12'] = df['v03'].apply(lambda x: ['p1', 'p2'][np.random.randint(2)])
    return df


def test1b():
    cols = 13
    rows = 10000000
    raw_data = np.random.randint(2, size=(rows,cols))
    col_names = ['v01', 'v02', 'v03', 'v04', 'v05', 'v06', 'v07',
                 'v08', 'v09', 'v10', 'v11', 'v12', 'outcome']
    df = pd.DataFrame(raw_data, columns=col_names)
    df['v11'] = np.take(
        np.array(['t1', 't2', 't3', 't4'], dtype=object),
        np.random.randint(4, size=rows))
    df['v12'] = np.take(
        np.array(['p1', 'p2'], dtype=object),
        np.random.randint(2, size=rows))
    return df


start_time = time.time()
test1a()
t1a = time.time() - start_time

start_time = time.time()
test1b()
t1b = time.time() - start_time

print("Test1a: {}sec, Test1b: {}sec".format(t1a, t1b))

Answer 1

让你慢下来的是pandasapply函数。 你可以用ipython%timeit函数来分析它，只是比较一下

%timeit df['v11'] = df['v03'].apply(lambda x: ['t1', 't2', 't3', 't4'][np.random.randint(4)])

至

%timeit df['v11'] = np.take(
    np.array(['t1', 't2', 't3', 't4'], dtype=object),
    np.random.randint(4, size=rows))

最终pandas.apply无法像numpy实现那样对代码进行矢量化，并导致在每次迭代中计算数据类型和重新调用python解释器时产生大量开销