擅长:python、mysql、java
<p>是的,你说得对。有一些虫子</p>
<p>我是说:</p>
<pre><code>def processLine(line):
#process something
print "result"
pool = Pool(processes = 8)
if __name__ == '__main__':
for line in sys.stdin:
lines.append(line)
if len(lines) >= 100000:
pool.map_async(processLine, lines, 2000)
lines = [] #to clear buffer
pool.map_async(processLine, lines, 2000)
pool.close()
pool.join()
</code></pre>
<p>我使用map_async是因为它有可配置的块大小,所以如果有很多处理时间很短的行,它会更高效。</p>