<p>这种情况下,<a href="http://dask.pydata.org/en/latest/dataframe.html" rel="nofollow">^{<cd1>}</a>可能会有所帮助。下面是一个例子。在</p>
<p>有关更多信息,请参见<a href="http://dask.pydata.org/en/latest/index.html" rel="nofollow">dask documentation</a>和这个非常漂亮的<a href="https://github.com/blaze/dask-tutorial" rel="nofollow">tutorial</a>。在</p>
<pre><code>In [1]: import dask.dataframe as dd
In [2]: !head data/accounts.0.csv
id,names,amount
138,Ray,1502
1,Tim,5
388,Ingrid,45
202,Zelda,1324
336,Jerry,-1607
456,Laura,-2832
65,Laura,-326
54,Yvonne,341
92,Sarah,3136
In [3]: dask_df = dd.read_csv('data/accounts.0.csv', chunkbytes=4000000)
In [4]: dask_df.npartitions
Out[4]: 4
In [5]: len(dask_df)
Out[5]: 1000000
In [6]: result = dask_df.groupby('names').sum()
In [7]: result.compute()
Out[7]:
id amount
names
Alice 10282524 43233084
Bob 8617531 47512276
Charlie 8056803 47729638
Dan 10146581 32513817
Edith 15164281 37806024
Frank 11310157 63869156
George 14941235 80436603
Hannah 3006336 25721574
Ingrid 10123877 54152865
Jerry 10317245 8613040
Kevin 6809100 16755317
Laura 9941112 34723055
Michael 11200937 36431387
Norbert 5715799 14482698
Oliver 10423117 32415534
Patricia 15289085 22767501
Quinn 10686459 16083432
Ray 10156429 9455663
Sarah 7977036 34970428
Tim 12283567 47851141
Ursula 4893696 37942347
Victor 8864468 15542688
Wendy 9348077 68824579
Xavier 6600945 -3482124
Yvonne 5665415 12701550
Zelda 8491817 42573021
</code></pre>
<p>这里是使用<code>pandas</code>进行比较的结果。我在这里使用的数据适合内存,但是<code>dask</code>即使数据大于内存也可以工作。在</p>
^{pr2}$