我有一个python线程问题。我已经四处找了一天多了,情况没有好转,所以我想寻求帮助。我使用python3.4
。
第一个问题是:
class myThread (threading.Thread):
def __init__(self, url):
threading.Thread.__init__(self)
self.url = url
def run(self):
spider (url)
我在代码的某个部分使用了toBeProcessed +'/robots.txt'
。如果我使用上述方法,它不会给我错误-但它仍然不能像它应该的那样工作,不是所有的线程都运行。如果我使用下面的方法,它会告诉我unsupported operand type(s) for +: '_thread._local' and 'str'
:
def run(self):
spider (self.url)
注意,我确实有这个声明toBeProcessed = threading.local()
。你知道吗
第二个问题是关于代码的其余部分,只有两个线程完成了工作,其余的线程——不管它们的编号是多少都不起作用。你知道吗
完整代码:
def spider(url,superMaxPages):
print(threading.current_thread())
toBeProcessed = threading.local()
data = threading.local()
parser = threading.local()
links = threading.local()
lock = threading.Lock()
writeLock = threading.Lock()
# Start from the beginning of our collection of pages to visit:
while 1:
if LinkParser.numVisited > maxPages:
print ('max pages reached')
break
lock.acquire()
try:
if not url:
time.sleep(0.01)
lock.release()
continue
print('to be processed ')
toBeProcessed = url.pop()
except:
print('threading error')
lock.release()
# In case we are not allowed to read the page.
rp = robotparser.RobotFileParser()
rp.set_url(toBeProcessed +'/robots.txt')
rp.read()
if not(rp.can_fetch("*", toBeProcessed)):
continue
LinkParser.visited.append(toBeProcessed)
LinkParser.numVisited += 1
writeLock.acquire()
try:
f.write(toBeProcessed+'\n')
finally:
writeLock.release()
try:
parser = LinkParser()
data, links = parser.getLinks(toBeProcessed)
# Add the pages that we visited to the end of our collection
url = url + links
print("One more page added from &i",threading.get_ident())
except:
print(" **Failed!**")
class myThread (threading.Thread):
def __init__(self, url, maxPages):
threading.Thread.__init__(self)
self.maxPages = maxPages
self.url = url
def run(self):
spider (self.url, maxPages)
不是像这样初始化url url = []
这就是我运行线程的方式,
myThread( spider, (url,maxPages) ).start
你做错了。不能用字符串连接^{} 的实例。您需要在
local
实例上存储一个属性,如下所示:此外,您应该考虑在} 。你知道吗
__init__
方法中使用^{相关问题 更多 >
编程相关推荐