我对使用python进行web抓取还是个新手,所以我不知道我是否做得对。在
我正在使用一个调用beauthoulsoup的脚本来解析google搜索的前10个页面的url。用stackoverflow.com网站,开箱即用。我在另一个网站上做了几次测试,试图看看这个脚本是否真的能处理更高的google页面请求,然后它就对我产生了503次的影响。我切换到另一个URL进行测试,并为几个低页面请求工作,然后也是503'd。现在我传递给它的每个URL都是503'ing。有什么建议吗?在
import sys # Used to add the BeautifulSoup folder the import path
import urllib2 # Used to read the html document
if __name__ == "__main__":
### Import Beautiful Soup
### Here, I have the BeautifulSoup folder in the level of this Python script
### So I need to tell Python where to look.
sys.path.append("./BeautifulSoup")
from BeautifulSoup import BeautifulSoup
### Create opener with Google-friendly user agent
opener = urllib2.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')]
### Open page & generate soup
### the "start" variable will be used to iterate through 10 pages.
for start in range(0,10):
url = "http://www.google.com/search?q=site:stackoverflow.com&start=" + str(start*10)
page = opener.open(url)
soup = BeautifulSoup(page)
### Parse and find
### Looks like google contains URLs in <cite> tags.
### So for each cite tag on each page (10), print its contents (url)
for cite in soup.findAll('cite'):
print cite.text
正如埃托雷所说,删除搜索结果是违反我们的任务规定的。但是,请查看websearchapi,特别是documentation的底部部分,它应该会告诉您如何从非javascip环境访问api。在
谷歌服务条款不允许自动查询。 有关信息,请参阅本文: Unusual traffic from your computer 还有Google Terms of service
相关问题 更多 >
编程相关推荐