用python从ajax请求中获取XML响应

Question

我试图在点击max（时间范围）按钮时获取加载到this page图表中的数据。数据通过ajax请求加载。你知道吗

我检查了请求，并试图用请求python库重现它，但我只能从这个图表中检索1年的数据。你知道吗

以下是我使用的代码：

r = requests.get("https://www.justetf.com/en/etf-profile.html?0-4.0-tabs-panel-chart-dates-ptl_max&groupField=none&sortField=ter&sortOrder=asc&from=search&isin=IE00B3VWN518&tab=chart&_=1576272593482")
r.content

我还尝试使用会话：

from requests import Session
session = Session()

session.head('http://justetf.com')

response = session.get(
    url='https://www.justetf.com/en/etf-profile.html?0-4.0-tabs-panel-chart-dates-ptl_max&groupField=none&sortField=ter&sortOrder=asc&from=search&isin=IE00B3VWN518&tab=chart&_=1575929227619',
    data = {"0-4.0-tabs-panel-chart-dates-ptl_max":"",
            "groupField":"none","sortField":"ter",
            "sortOrder":"asc","from":"search",
            "isin":"IE00B3VWN518",
            "tab":"chart",
            "_":"1575929227619"
           },

    headers={
        'Host': 'www.justetf.com',
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0',
        'Accept': 'application/xml, text/xml, */*; q=0.01',
        'Accept-Language': 'en-US,en;q=0.5',
        'Accept-Encoding': 'gzip, deflate, br',
        'Wicket-Ajax': 'true',
        'Wicket-Ajax-BaseURL': 'en/etf-profile.html?0&groupField=none&sortField=ter&sortOrder=asc&from=search&isin=IE00B3VWN518&tab=chart',
        'Wicket-FocusedElementId': 'id28',
        'X-Requested-With': 'XMLHttpRequest',
        'Connection': 'keep-alive',
        'Referer': 'https://www.justetf.com/en/etf-profile.html?groupField=none&sortField=ter&sortOrder=asc&from=search&isin=IE00B3VWN518&tab=chart',
        'Cookie': 'locale_=en; _ga=GA1.2.1297456970.1574289342; cookieconsent_status=dismiss; AWSALB=QMWHJxgfcpLXJLqX0i0FgBuLn+mpVHVeLRQ6upH338LdggA4/thXHT2vVWQX7pdBd1r486usZXgpAF8RpDsGJNtf6ei8e5NHTsg0hzVHR9C+Fj89AWuQ7ue+fzV2; JSESSIONID=ABB2A35B91751CA9B2D293F5A04505BE; _gid=GA1.2.1029531470.1575928527; _gat=1',
        'TE': 'Trailer'


        },

    cookies = {"_ga":"GA1.2.1297456970.1574289342","_gid":"GA1.2.1411779365.1574289342","AWSALB":"5v+tPMgooQC0deJBlEGl2wVeUSmwVGJdydie1D6dAZSRAK5eBsmg+DQCdBj8t25YRytC5NIi0TbU3PmDcNMjiyFPTp1xKHgwNjZcDvMRePZjTxthds5DsvelzE2I","JSESSIONID":"310F346AED94D1A345207A3489DCF83D","locale_":"en"}
)

但我得到了这样的回应

<ajax-response><redirect><![CDATA[/en/etf-profile.html?0&groupField=none&sortField=ter&sortOrder=asc&from=search&isin=IE00B3VWN518&tab=chart]]></redirect></ajax-response>

为什么当我点击MAX时，浏览器上的XML文件没有得到响应？你知道吗

Answer 1

好的，下面是我获取您所需数据的解决方案：

url = "https://www.justetf.com/en/etf-profile.html"

querystring = {
  # Modify this string to get the timeline you want
  # Currently it is set to "max" as you can see
  "0-1.0-tabs-panel-chart-dates-ptl_max":"",
  "groupField":"none",
  "sortField":"ter",
  "sortOrder":"asc",
  "from":"search",
  "isin":"IE00B3VWN518",
  "tab":"chart",
  "_":"1576627890798"}

# Not all of these headers may be necessary
headers = {
    'authority': "www.justetf.com",
    'accept': "application/xml, text/xml, */*; q=0.01",
    'x-requested-with': "XMLHttpRequest",
    'wicket-ajax-baseurl': "en/etf-profile.html?0&groupField=none&sortField=ter&sortOrder=asc&from=search&isin=IE00B3VWN518&tab=chart",
    'wicket-ajax': "true",
    'wicket-focusedelementid': "id27",
    'Connection': "keep-alive",
}

session = requests.Session()

# The first request won't return what we want but it sets the cookies
response = session.get( url, params=querystring)

# Cookies have been set now we can make the 2nd request and get the data we want
response = session.get( url, headers=headers, params=querystring)

print(response.text)

作为奖励，我加入了一个链接到复制在这里我实际解析数据并得到每个单独的数据点。你可以找到这个here。你知道吗

如果有用的话就告诉我！你知道吗