<p>这通常取决于问题的大小</p>
<p>让我们做一些调查</p>
<p><strong>结论:</strong></p>
<p>Numpy阵列比即使不使用Numpy函数也要快</p>
<p>numpy函数的使用仍在加速,但代价是:</p>
<ul>
<li>一些初始化时间</li>
<li>导致警告的精度损失</li>
</ul>
<h2>调查</h2>
<h2>初始化</h2>
<pre><code>from collections import namedtuple
import pandas as pd
import numpy as np
import math
</code></pre>
<pre><code>Run = namedtuple('run', ['order', 'npa', 'df'])
def generate(maxorder):
run_list = []
for n in range(1, maxorder + 1):
order = n
npa = np.random.random(size=(10**n, 3))
run_list.append(Run(order, npa, pd.DataFrame(npa, columns=list('xyz'))))
return run_list
</code></pre>
<pre><code>runs = generate(5)
</code></pre>
<h2>工具</h2>
<pre><code>from functools import wraps
import time
</code></pre>
<pre><code>def timeit(func):
@wraps(func)
def timed(*args, **kw):
start_time = time.time()
result = func(*args, **kw)
end_time = time.time()
print(f"{func.__name__} {(end_time - start_time) * 1000} ms")
return result
return timed
</code></pre>
<pre><code>def prints(f, data, max_order):
for order,npa,df in runs[:max_order]:
print()
print("Order:", order)
maxGrad, maxCoorPair = f(df if data == 'df' else npa)
print("maxGrad:", maxGrad)
print("Pair:")
print(maxCoorPair)
</code></pre>
<h2>解决方案</h2>
<h3>原始溶液</h3>
<pre><code>@timeit
def sol_original(df):
maxGrad = 0
currentGrad = 0
height = 0
for i in range(len(df)):
for j in range(i+1,len(df)):
height = abs(df.z.iloc[j]-df.z.iloc[i])
distance = math.sqrt((df.x.iloc[j]-df.x.iloc[i])**2+(df.y.iloc[j]-df.y.iloc[i])**2)
currentGrad = height/distance
if currentGrad > maxGrad:
maxGrad = currentGrad
maxCoorPair = [(df.x.iloc[i],df.y.iloc[i]),(df.x.iloc[j],df.y.iloc[j])]
return maxGrad, maxCoorPair
</code></pre>
<pre><code>prints(sol_original, 'df', 3)
</code></pre>
<pre><code>Order: 1
sol_original 2.979278564453125 ms
maxGrad: 4.274082602280762
Pair:
[(0.08217955028694601, 0.9160537098844143), (0.2396284679279188, 0.8196073645585937)]
Order: 2
sol_original 264.29247856140137 ms
maxGrad: 167.5282986999116
Pair:
[(0.09926115331767238, 0.7497707080285022), (0.09615387652529517, 0.7469231082687531)]
Order: 3
sol_original 24213.2625579834 ms
maxGrad: 1246.4073209631038
Pair:
[(0.8285207768494603, 0.016839864860434428), (0.8285102753052541, 0.016228726039262287)]
</code></pre>
<h3>解决方案numpy1.0:将np数组替换为df</h3>
<pre><code>@timeit
def sol_numpy10(npa):
maxGrad = 0
currentGrad = 0
height = 0
for i in range(len(npa)):
for j in range(i+1,len(npa)):
height = abs(npa[j][2]-npa[i][2])
distance = math.sqrt((npa[j][0]-npa[i][0])**2+(npa[j][1]-npa[i][1])**2)
currentGrad = height/distance
if currentGrad > maxGrad:
maxGrad = currentGrad
maxCoorPair = [(npa[i][0],npa[i][1]),(npa[j][0],npa[j][1])]
return maxGrad, maxCoorPair
</code></pre>
<pre><code>prints(sol_numpy10, 'np', 3)
</code></pre>
<pre><code>Order: 1
sol_numpy10 0.0 ms
maxGrad: 4.274082602280762
Pair:
[(0.08217955028694601, 0.9160537098844143), (0.2396284679279188, 0.8196073645585937)]
Order: 2
sol_numpy10 13.963699340820312 ms
maxGrad: 167.5282986999116
Pair:
[(0.09926115331767238, 0.7497707080285022), (0.09615387652529517, 0.7469231082687531)]
Order: 3
sol_numpy10 998.3282089233398 ms
maxGrad: 1246.4073209631038
Pair:
[(0.8285207768494603, 0.016839864860434428), (0.8285102753052541, 0.016228726039262287)]
</code></pre>
<h3>解决方案numpy1.1:idem numpy1.0并优化i坐标</h3>
<pre><code>@timeit
def sol_numpy11(npa):
maxGrad = 0
currentGrad = 0
height = 0
for i in range(len(npa)):
xi,yi,zi = npa[i]
for j in range(i+1,len(npa)):
height = abs(npa[j][2]-zi)
distance = math.sqrt((npa[j][0]-xi)**2+(npa[j][1]-yi)**2)
currentGrad = height/distance
if currentGrad > maxGrad:
maxGrad = currentGrad
maxCoorPair = [(xi,yi),(npa[j][0],npa[j][1])]
return(maxGrad, maxCoorPair)
</code></pre>
<pre><code>prints(sol_numpy11, 'np', 3)
</code></pre>
<pre><code>Order: 1
sol_numpy11 0.0 ms
maxGrad: 4.274082602280762
Pair:
[(0.08217955028694601, 0.9160537098844143), (0.2396284679279188, 0.8196073645585937)]
Order: 2
sol_numpy11 10.002613067626953 ms
maxGrad: 167.5282986999116
Pair:
[(0.09926115331767238, 0.7497707080285022), (0.09615387652529517, 0.7469231082687531)]
Order: 3
sol_numpy11 771.9049453735352 ms
maxGrad: 1246.4073209631038
Pair:
[(0.8285207768494603, 0.016839864860434428), (0.8285102753052541, 0.016228726039262287)]
</code></pre>
<h3>解决方案numpy1.2;idem numpy1.1并优化j坐标</h3>
<pre><code>@timeit
def sol_numpy12(npa):
maxGrad = 0
currentGrad = 0
height = 0
for i in range(len(npa)):
xi,yi,zi = npa[i]
for j in range(i+1,len(npa)):
xj,yj,zj = npa[j]
height = abs(zj-zi)
distance = math.sqrt((xj-xi)**2+(yj-yi)**2)
currentGrad = height/distance
if currentGrad > maxGrad:
maxGrad = currentGrad
maxCoorPair = [(xi,yi),(xj,yj)]
return(maxGrad, maxCoorPair)
</code></pre>
<pre><code>prints(sol_numpy12, 'np', 3)
</code></pre>
<pre><code>Order: 1
sol_numpy12 0.0 ms
maxGrad: 4.274082602280762
Pair:
[(0.08217955028694601, 0.9160537098844143), (0.2396284679279188, 0.8196073645585937)]
Order: 2
sol_numpy12 8.97526741027832 ms
maxGrad: 167.5282986999116
Pair:
[(0.09926115331767238, 0.7497707080285022), (0.09615387652529517, 0.7469231082687531)]
Order: 3
sol_numpy12 1075.1206874847412 ms
maxGrad: 1246.4073209631038
Pair:
[(0.8285207768494603, 0.016839864860434428), (0.8285102753052541, 0.016228726039262287)]
</code></pre>
<h3>解决方案numpy2.0</h3>
<p>莱利的回答</p>
<pre><code>def d(vector):
return vector.apply(lambda x: x-vector).values
@timeit
def sol_numpy20(df):
slopes = np.abs(d(df["z"]))/np.sqrt(np.square(d(df["x"])) + np.square(d(df["y"])))
maxGrad = np.nanmax(slopes)
ind1, ind2 = np.where(slopes==maxGrad)
maxCoorPair = [df.iloc[ind2]]
return maxGrad, maxCoorPair
</code></pre>
<pre><code>prints(sol_numpy20, 'df', 4)
</code></pre>
<pre><code>Order: 1
10
sol_numpy20 6.981372833251953 ms
maxGrad: 4.274082602280762
Pair:
[ x y z
6 0.239628 0.819607 0.867972
5 0.082180 0.916054 0.078804]
Order: 2
100
sol_numpy20 42.88601875305176 ms
maxGrad: 167.5282986999116
Pair:
[ x y z
95 0.096154 0.746923 0.135159
45 0.099261 0.749771 0.841247]
Order: 3
1000
<ipython-input-201-ce6f8e091e1b>:7: RuntimeWarning: invalid value encountered in true_divide
slopes = np.abs(d(df["z"]))/np.sqrt(np.square(d(df["x"])) + np.square(d(df["y"])))
<ipython-input-201-ce6f8e091e1b>:7: RuntimeWarning: invalid value encountered in true_divide
slopes = np.abs(d(df["z"]))/np.sqrt(np.square(d(df["x"])) + np.square(d(df["y"])))
<ipython-input-201-ce6f8e091e1b>:7: RuntimeWarning: invalid value encountered in true_divide
slopes = np.abs(d(df["z"]))/np.sqrt(np.square(d(df["x"])) + np.square(d(df["y"])))
sol_numpy20 625.3554821014404 ms
maxGrad: 1246.4073209631038
Pair:
[ x y z
991 0.828510 0.016229 0.893811
240 0.828521 0.016840 0.131971]
Order: 4
10000
<ipython-input-201-ce6f8e091e1b>:7: RuntimeWarning: invalid value encountered in true_divide
slopes = np.abs(d(df["z"]))/np.sqrt(np.square(d(df["x"])) + np.square(d(df["y"])))
sol_numpy20 12746.904850006104 ms
maxGrad: 4346.72025021119
Pair:
[ x y z
3751 0.538723 0.168160 0.131924
1063 0.538800 0.168028 0.798256]
</code></pre>
<pre><code>
</code></pre>