如何加速一个热编码器cod

Question

我做了一个简单的函数，当输入一个向量时，它将返回一个输出的一个热编码矩阵。在

import numpy as np

def ohc(x):
    u = list(set(x))
    c = len(u)
    X = np.zeros((len(x), c))
    for idx, val in enumerate(x):
        for i in range(c):
            if val == u[i]:
                X[idx, i] = 1
    return X

inputx = np.random.randint(1, 4, 1000000)
ohc(inputx) 
Out[2]: 
array([[ 0.,  1.,  0.],
       [ 0.,  1.,  0.],
       [ 0.,  1.,  0.],
       ..., 
       [ 0.,  0.,  1.],
       [ 0.,  1.,  0.],
       [ 0.,  1.,  0.]])

但我想知道是否因为这两个for循环，有没有办法加快速度？在

^{pr2}$

Answer 1

这里有一个矢量化方法，只使用^{}中的唯一值，与原始数组进行比较，从而得到一个热编码数组-

(inputx[:,None] == np.unique(inputx)).astype(float)

运行时测试

其他方法-

^{pr2}$

时间安排-

In [42]: inputx = np.random.randint(1, 4, 1000000)

In [43]: %timeit ohc(inputx)
1 loops, best of 3: 526 ms per loop

In [44]: %timeit ohc_dict(inputx)
1 loops, best of 3: 256 ms per loop

In [45]: %timeit unique_inverse(inputx)
10 loops, best of 3: 48.6 ms per loop

In [46]: %timeit (inputx[:,None] == np.unique(inputx)).astype(float)
10 loops, best of 3: 34.4 ms per loop

进一步提高性能-

使用np.int8作为输出数据类型，以便通过建议的方法进一步提高性能-

In [58]: %timeit (inputx[:,None] == np.unique(inputx)).astype(np.int8)
10 loops, best of 3: 27.7 ms per loop

正如@paulpanzer建议的那样，我们也可以使用view来代替类型转换，以进一步增强数组的唯一性-

In [23]: inputx = np.random.randint(1, 40, 1000000)

In [24]: %timeit (inputx[:,None] == np.unique(inputx)).astype(np.int8)
10 loops, best of 3: 98.4 ms per loop

In [25]: %timeit (inputx[:,None] == np.unique(inputx)).view(np.int8)
10 loops, best of 3: 92.5 ms per loop

Answer 2

您的代码在O（n[因为set（）]+nc[因为for循环]）。在大多数实际应用程序中，最终都会得到O（nc）*，因为您需要为数组分配空间。不过，有几个技巧可以让它更高效：

1. 使用dict。Dicts是使用散列实现的，平均来说，散列时间应该是恒定的。在
2. 不要在每一步迭代c可能的特性，而是记住每个特性的索引。在

以下是我的实现：

import numpy as np

def ohc(x):
    elem_to_idx = {}
    for e in x:
        if e not in elem_to_idx:
            elem_to_idx[e] = len(elem_to_idx)
    c = len(elem_to_idx)
    X = np.zeros((len(x), c))
    for idx, val in enumerate(x):
        X[idx, elem_to_idx[val]] = 1
    return X

*根据您打算如何处理X矩阵，您可能需要使用numpy.稀疏你的矩阵不能分配很多内存

Answer 3

看起来是np.unique的工作

uniq, inv = np.unique(x, return_inverse=True)
result = np.zeros((len(x), len(uniq)), dtype=int)
result[np.arange(len(x)), inv] = 1

针对@Divakar的基准测试：这里有一个信息更丰富的比较，证实了dv在小字母表中的一个轻微的速度优势，它在K=20附近交叉，在K=1000处，它又反过来成为{}的几倍优势。这是预期的，因为pp利用了一个热的稀疏性。下面，K是字母表的大小，N是样本的长度。在

^{pr2}$

印刷品：

@ K = 4
dv: 0.003458, 0.038176, 0.421894 secs for 100 trials @ N = 1000, 10000, 100000
pp: 0.004856, 0.052298, 0.603758 secs for 100 trials @ N = 1000, 10000, 100000
@ K = 10
dv: 0.005136, 0.056491, 0.663157 secs for 100 trials @ N = 1000, 10000, 100000
pp: 0.005955, 0.054069, 0.719152 secs for 100 trials @ N = 1000, 10000, 100000
@ K = 20
dv: 0.007201, 0.084867, 0.988886 secs for 100 trials @ N = 1000, 10000, 100000
pp: 0.007638, 0.084580, 0.891122 secs for 100 trials @ N = 1000, 10000, 100000
@ K = 40
dv: 0.010748, 0.130974, 1.498022 secs for 100 trials @ N = 1000, 10000, 100000
pp: 0.009321, 0.103912, 1.080271 secs for 100 trials @ N = 1000, 10000, 100000
@ K = 100
dv: 0.025357, 0.292930, 2.946326 secs for 100 trials @ N = 1000, 10000, 100000
pp: 0.011916, 0.147117, 1.641588 secs for 100 trials @ N = 1000, 10000, 100000
@ K = 200
dv: 0.033651, 0.560753, 6.042001 secs for 100 trials @ N = 1000, 10000, 100000
pp: 0.022971, 0.221142, 3.580255 secs for 100 trials @ N = 1000, 10000, 100000
@ K = 1000
dv: 0.156715, 2.655647, 37.112166 secs for 100 trials @ N = 1000, 10000, 100000
pp: 0.055516, 0.920938, 10.358050 secs for 100 trials @ N = 1000, 10000, 100000

使用uint8并允许@Divakar的方法使用更便宜的视图转换：

@ K = 4
dv: 0.003092, 0.038149, 0.386140 secs for 100 trials @ N = 1000, 10000, 100000
pp: 0.004392, 0.043327, 0.554253 secs for 100 trials @ N = 1000, 10000, 100000
@ K = 10
dv: 0.004604, 0.054215, 0.501708 secs for 100 trials @ N = 1000, 10000, 100000
pp: 0.004930, 0.051555, 0.607239 secs for 100 trials @ N = 1000, 10000, 100000
@ K = 20
dv: 0.006421, 0.067397, 0.665465 secs for 100 trials @ N = 1000, 10000, 100000
pp: 0.006616, 0.054055, 0.703260 secs for 100 trials @ N = 1000, 10000, 100000
@ K = 40
dv: 0.008857, 0.087155, 0.862316 secs for 100 trials @ N = 1000, 10000, 100000
pp: 0.006945, 0.060408, 0.733966 secs for 100 trials @ N = 1000, 10000, 100000
@ K = 100
dv: 0.015660, 0.142464, 1.426929 secs for 100 trials @ N = 1000, 10000, 100000
pp: 0.008063, 0.070860, 0.908615 secs for 100 trials @ N = 1000, 10000, 100000
@ K = 200
dv: 0.025631, 0.235712, 2.401750 secs for 100 trials @ N = 1000, 10000, 100000
pp: 0.008805, 0.101772, 1.111652 secs for 100 trials @ N = 1000, 10000, 100000
@ K = 1000
dv: 0.069953, 1.024585, 11.313402 secs for 100 trials @ N = 1000, 10000, 100000
pp: 0.011558, 0.182684, 2.201837 secs for 100 trials @ N = 1000, 10000, 100000