我试图构建一个类,它使用multiprocessing+请求并行地发出多个请求。我遇到了一个问题,它只是挂起,并给我一个神秘的错误消息,我不确定的方式。在
下面是我的代码,它基本上只是使用一个带有回调函数的池来将结果放入列表中。我有一个要求,我需要一个“硬超时”为每个网址,也就是说,如果一个网址需要超过几秒钟的时间来下载其内容,我只想跳过它。所以我使用了一个池超时,并对尝试的URL和返回的URL内容进行了比较,尝试但没有返回的内容被假定为失败。这是我的代码:
import time
import json
import requests
import sys
from urlparse import parse_qs
from urlparse import urlparse
from urlparse import urlunparse
from urllib import urlencode
from multiprocessing import Process, Pool, Queue, current_process
from multiprocessing.pool import ThreadPool
from multiprocessing import TimeoutError
import traceback
from sets import Set
from massweb.pnk_net.pnk_request import pnk_request_raw
from massweb.targets.fuzzy_target import FuzzyTarget
from massweb.payloads.payload import Payload
class MassRequest(object):
def __init__(self, num_threads = 10, time_per_url = 10, request_timeout = 10, proxy_list = [{}]):
self.num_threads = num_threads
self.time_per_url = time_per_url
self.request_timeout = request_timeout
self.proxy_list = proxy_list
self.results = []
self.urls_finished = []
self.urls_attempted = []
self.targets_results = []
self.targets_finished = []
self.targets_attempted = []
def add_to_finished(self, x):
self.urls_finished.append(x[0])
self.results.append(x)
def add_to_finished_targets(self, x):
self.targets_finished.append(x[0])
self.targets_results.append(x)
def get_urls(self, urls):
timeout = float(self.time_per_url * len(urls))
pool = Pool(processes = self.num_threads)
proc_results = []
for url in urls:
self.urls_attempted.append(url)
proc_result = pool.apply_async(func = pnk_request_raw, args = (url, self.request_timeout, self.proxy_list), callback = self.add_to_finished)
proc_results.append(proc_result)
for pr in proc_results:
try:
pr.get(timeout = timeout)
except:
pool.terminate()
pool.join()
pool.terminate()
pool.join()
list_diff = Set(self.urls_attempted).difference(Set(self.urls_finished))
for url in list_diff:
sys.stderr.write("URL %s got timeout" % url)
self.results.append((url, "__PNK_GET_THREAD_TIMEOUT"))
if __name__ == "__main__":
f = open("out_urls_to_fuzz_1mil")
urls_to_request = []
for line in f:
url = line.strip()
urls_to_request.append(url)
mr = MassRequest()
mr.get_urls(urls_to_request)
以下是线程调用的函数:
^{pr2}$这对于较小的URL集似乎很有效,但以下是输出:
Requesting: http://www.sportspix.co.za/ with proxy {}
Requesting: http://www.sportspool.co.za/ with proxy {}
Requesting: http://www.sportspredict.co.za/ with proxy {}
Requesting: http://www.sportspro.co.za/ with proxy {}
Requesting: http://www.sportsrun.co.za/ with proxy {}
Requesting: http://www.sportsstuff.co.za/ with proxy {}
Requesting: http://sportsstuff.co.za/2011-rugby-world-cup with proxy {}
Requesting: http://www.sportstar.co.za/4-stroke-racing with proxy {}
Requesting: http://www.sportstats.co.za/ with proxy {}
Requesting: http://www.sportsteam.co.za/ with proxy {}
Requesting: http://www.sportstec.co.za/ with proxy {}
Requesting: http://www.sportstours.co.za/ with proxy {}
Requesting: http://www.sportstrader.co.za/ with proxy {}
Requesting: http://www.sportstravel.co.za/ with proxy {}
Requesting: http://www.sportsturf.co.za/ with proxy {}
Requesting: http://reimo.sportsvans.co.za/ with proxy {}
Requesting: http://www.sportsvans.co.za/4x4andmoreWindhoek.html with proxy {}
Handled exception:Traceback (most recent call last):
File "mass_request.py", line 87, in get_fuzzy_targets
pr.get(timeout = timeout)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 528, in get
raise self._value
AttributeError: 'file' object has no attribute 'out'
在最后一个异常情况下,程序挂起了,我必须完全终止它。我从来没有尝试过访问属性为“out”的file对象。我的问题是。。。怎么修好!?我在这里做了什么明显的错误吗?为什么没有更明确的例外?在
我认为}
sys.stderr.out("A request failed to URL %s\n" % url_or_target)
应该是{相关问题 更多 >
编程相关推荐