httplib没有得到所有的重定向代码

Question

我正在尝试获取一个页面的最终url，该页面似乎重定向了不止一次。在浏览器中尝试此示例URL，并将其与代码段底部的最终URL进行比较：

Link that redirects more than once

这是我运行的测试代码，注意最后得到代码200的URL与浏览器中的不同。我有什么选择？

Python 2.7.1+ (r271:86832, Apr 11 2011, 18:13:53)
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import httplib
>>> from urlparse import urlparse
>>> url = 'http://www.usmc.mil/units/hqmc/'
>>> host = urlparse(url)[1]
>>> req = ''.join(urlparse(url)[2:5])
>>> conn = httplib.HTTPConnection(host)
>>> conn.request('HEAD', req)
>>> resp = conn.getresponse()
>>> print resp.status
    301
>>> print resp.msg.dict['location']
    http://www.marines.mil/units/hqmc/

>>> url = 'http://www.marines.mil/units/hqmc/'
>>> host = urlparse(url)[1]
>>> req = ''.join(urlparse(url)[2:5])
>>> conn = httplib.HTTPConnection(host)
>>> conn.request('HEAD', req)
>>> resp = conn.getresponse()
>>> print resp.status
    302
>>> print resp.msg.dict['location']
    http://www.marines.mil/units/hqmc/default.aspx

>>> url = 'http://www.marines.mil/units/hqmc/default.aspx'
>>> host = urlparse(url)[1]
>>> req = ''.join(urlparse(url)[2:5])
>>> conn = httplib.HTTPConnection(host)
>>> conn.request('HEAD', req)
>>> resp = conn.getresponse()
>>> print resp.status
    200
>>> print resp.msg.dict['location']
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    KeyError: 'location'
>>> print url
    http://www.marines.mil/units/hqmc/default.aspx //THIS URL DOES NOT RETURN A 200 IN ANY BROWSER I HAVE TRIED 

Answer 1

您可以尝试将用户代理头设置为浏览器的用户代理。

附言： urllib2自动重定向

编辑：

In [2]: import urllib2
In [3]: resp = urllib2.urlopen('http://www.usmc.mil/units/hqmc/')
In [4]: resp.geturl()
Out[4]: 'http://www.marines.mil/units/hqmc/default.aspx