擅长:python、mysql、java
<p>很难说这是否以及有多大帮助(因为我没有什么要测试的…),但你可以试试<a href="https://docs.python.org/3/library/multiprocessing.html#module-multiprocessing.pool" rel="nofollow noreferrer">^{<cd1>}</a>。它为您处理所有脏活,您可以自定义进程数、块大小等</p>
<pre class="lang-py prettyprint-override"><code>from multiprocessing import Pool
def worker(x):
myData = DownloadData(x)
return ParseData(myData)
if __name__ == "__main__":
processes = None # defaults to os.cpu_count()
chunksize = 1
with Pool(processes) as pool:
parsedData = pool.map(worker, lastTenYears, chunksize)
</code></pre>
<p>在这里的示例中,我使用<a href="https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.map" rel="nofollow noreferrer">^{<cd2>}</a>方法,但根据您的需要,您可能希望使用<code>imap</code>或<code>map_async</code></p>