当python只在浏览时显示重定向位置时，如何使用python获取重定向位置

Question

我正在尝试从http://www.pingshu8.com/down_121010.html下载一些音频剪辑

通过使用scrapy，我获得了真正的文件链接：

from urllib.parse import unquote
fetch('http://www.pingshu8.com/down_121008.html')
downurl=response.xpath('//script/text()').extract()[0]
downurl=unquote(unquote(downurl))
downurl1=response.urljoin(re.search('bzmtv_Inc/download.asp\?fid=\d+&t=\d+',downurl).group())

但当我使用

fetch(downurl1)

我得到了200个状态，而不是301重定向到正确的网址

我发现只有当我点击下载按钮时才能得到正确的结果

Answer 1

网站同时检查Referer和用户代理：

import re
from urllib import unquote
from scrapy.http import Request

initial_url = 'http://www.pingshu8.com/down_121008.html'
fetch(initial_url)
downurl = unquote(unquote(response.xpath('//script/text()').extract()[0]))
downurl = response.urljoin(re.search('bzmtv_Inc/download.asp\?fid=\d+&t=\d+', downurl).group())

# if you request without these required headers (User-Agent and Referer) no content is returned
r = Request(downurl)
fetch(r)
print response.body

# but if you request with those headers, it will return content
request_with_headers = Request(
      downurl,
      headers = {
        'Referer': initial_url,
        'User-Agent':'Mozilla/4.6 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.326.109 Safari/507.23',
      },
      meta = {
        'dont_redirect': True,
        'handle_httpstatus_list': [301,302],
      }
)
fetch(request_with_headers)
print response.headers.get('Location')

带有标题的示例输出：

>>> print response.headers.get('Location')
https://down01.pingshu8.com:8011/2/ys/儿童文学60年经典回放(16集)/儿童文学60年经典回放_01.mp3?t=ztmu1924596a0610b630ddd21a54fda258fa0&m=5C9208AE