<p>对于这样大的数据集,最有帮助的是理解如何以<code>pandas</code>的方式来做事情。与内置的矢量化方法相比,遍历行将产生糟糕的性能。在</p>
<pre><code>import pandas as pd
import numpy as np
df = pd.read_csv('/yellow_tripdata_2016-06.csv')
df.head(5)
VendorID tpep_pickup_datetime tpep_dropoff_datetime passenger_count trip_distance pickup_longitude pickup_latitude RatecodeID store_and_fwd_flag dropoff_longitude dropoff_latitude payment_type fare_amount extra mta_tax tip_amount tolls_amount improvement_surcharge total_amount
0 2 2016-06-09 21:06:36 2016-06-09 21:13:08 2 0.79 -73.983360 40.760937 1 N -73.977463 40.753979 2 6.0 0.5 0.5 0.00 0.0 0.3 7.30
1 2 2016-06-09 21:06:36 2016-06-09 21:35:11 1 5.22 -73.981720 40.736668 1 N -73.981636 40.670242 1 22.0 0.5 0.5 4.00 0.0 0.3 27.30
2 2 2016-06-09 21:06:36 2016-06-09 21:13:10 1 1.26 -73.994316 40.751072 1 N -74.004234 40.742168 1 6.5 0.5 0.5 1.56 0.0 0.3 9.36
3 2 2016-06-09 21:06:36 2016-06-09 21:36:10 1 7.39 -73.982361 40.773891 1 N -73.929466 40.851540 1 26.0 0.5 0.5 1.00 0.0 0.3 28.30
4 2 2016-06-09 21:06:36 2016-06-09 21:23:23 1 3.10 -73.987106 40.733173 1 N -73.985909 40.766445 1 13.5 0.5 0.5 2.96 0.0 0.3 17.76
</code></pre>
<p>这个数据集有11135470行,这不是“大数据”,但也不小。与其编写一个函数并将其应用于每一行,不如将函数的某些部分执行到各个列,从而获得更高的性能。我会把这个函数转过来:</p>
^{pr2}$
<p>在这方面:</p>
<pre><code>SEMIMAJORAXIS = 6378137.0 # typed in all caps since this is a static value
df['pickup_east'] = df['pickup_longitude'] * 0.017453292519943295 # takes all pickup longitude values, multiples them, then saves as a new column named pickup_east.
df['pickup_north'] = df['pickup_latitude'] * 0.017453292519943295
# numpy functions allow you to calculate an entire column's worth of values by simply passing in the column.
df['pickup_northing'] = 3189068.5 * np.log((1.0 + np.sin(df['pickup_north'])) / (1.0 - np.sin(df['pickup_north'])))
df['pickup_easting'] = SEMIMAJORAXIS * df['pickup_east']
</code></pre>
<p>然后,<code>pickup_easting</code>和{<cd3>}列包含计算值。在</p>
<p>对于我的笔记本电脑,这需要:</p>
<pre><code>CPU times: user 1.01 s, sys: 286 ms, total: 1.3 s
Wall time: 763 ms
</code></pre>
<p>所有1100万行。15分钟>秒。在</p>
<p>我取消了价值观的检查-你可以做些类似的事情:</p>
<pre><code>df = df[(df['pickup_longitude'].abs() <= 180) & (df['pickup_latitude'].abs() <= 90)]
</code></pre>
<p>这使用了布尔索引,这同样比循环快几个数量级。在</p>