<p>使用<a href="https://docs.python.org/3.4/library/multiprocessing.html?highlight=process" rel="nofollow noreferrer">multiprocessing</a>可以考虑使用<a href="https://docs.python.org/3.4/library/multiprocessing.html?highlight=process#multiprocessing.Queue" rel="nofollow noreferrer">Queue</a>。你知道吗</p>
<p>通常您会创建两个作业,一个创建URL,另一个使用URL。我们叫它们<code>creator</code>和<code>consumer</code>。我将假设这里的任何信号量都被称为<code>closing_condition</code>(例如使用<a href="https://docs.python.org/3.4/library/multiprocessing.html?highlight=process#multiprocessing.Value" rel="nofollow noreferrer">Value</a>),用于解析url并保存它们的方法分别被称为<code>create_url_method</code>和<code>store_url</code>。你知道吗</p>
<pre><code>from multiprocessing import Queue, Value, Process
import queue
def creator(urls, closing_condition):
"""Parse page and put urls in given Queue."""
while (not closing_condition):
created_urls = create_url_method()
[urls.put(url) for url in created_urls]
def consumer(urls, closing_condition):
"""Consume urls in given Queue."""
while (not closing_condition):
try:
store_url(urls.get(timeout=1))
except queue.Empty:
pass
urls = Queue()
semaphore = Value('d', 0)
creators_number = 2
consumers_number = 2
creators = [
Process(target=creator, args=(urls, semaphore))
for i in range(creators_number)
]
consumers = [
Process(target=consumer, args=(urls, semaphore))
for i in range(consumers_number)
]
[p.start() for p in creators + consumer]
[p.join() for p in creators + consumer]
</code></pre>