我正在尝试获取一个页面的最终url,该页面似乎重定向了不止一次。在浏览器中尝试此示例URL,并将其与代码段底部的最终URL进行比较:
Link that redirects more than once
这是我运行的测试代码,注意最后得到代码200的URL与浏览器中的不同。我有什么选择?
Python 2.7.1+ (r271:86832, Apr 11 2011, 18:13:53)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import httplib
>>> from urlparse import urlparse
>>> url = 'http://www.usmc.mil/units/hqmc/'
>>> host = urlparse(url)[1]
>>> req = ''.join(urlparse(url)[2:5])
>>> conn = httplib.HTTPConnection(host)
>>> conn.request('HEAD', req)
>>> resp = conn.getresponse()
>>> print resp.status
301
>>> print resp.msg.dict['location']
http://www.marines.mil/units/hqmc/
>>> url = 'http://www.marines.mil/units/hqmc/'
>>> host = urlparse(url)[1]
>>> req = ''.join(urlparse(url)[2:5])
>>> conn = httplib.HTTPConnection(host)
>>> conn.request('HEAD', req)
>>> resp = conn.getresponse()
>>> print resp.status
302
>>> print resp.msg.dict['location']
http://www.marines.mil/units/hqmc/default.aspx
>>> url = 'http://www.marines.mil/units/hqmc/default.aspx'
>>> host = urlparse(url)[1]
>>> req = ''.join(urlparse(url)[2:5])
>>> conn = httplib.HTTPConnection(host)
>>> conn.request('HEAD', req)
>>> resp = conn.getresponse()
>>> print resp.status
200
>>> print resp.msg.dict['location']
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
KeyError: 'location'
>>> print url
http://www.marines.mil/units/hqmc/default.aspx //THIS URL DOES NOT RETURN A 200 IN ANY BROWSER I HAVE TRIED
您可以使用HttpLib2获取URL的实际位置:
执行过程如下:
注意这个例子也使用缓存(urllib和httplib都不支持缓存)。因此,这将反复明显加快运行速度。这可能对爬行/抓取很有意思。如果不需要缓存,请将
h = httplib2.Http(".cache_httplib")
替换为h = httplib2.Http()
。您可以尝试将用户代理头设置为浏览器的用户代理。
附言: urllib2自动重定向
编辑:
相关问题 更多 >
编程相关推荐