python请求不返回json数据

Question

我想使用python请求模块从实例https://app.weathercloud.net/d0838117883#current获取json数据。 我试过：

    import re
    import requests
    
    device='0838117883'
    URL='https://app.weathercloud.net'
    URL1=URL+'/d'+device
    URL2=URL+'/device/stats'
    
    headers={'Content-Type':'text/plain; charset=UTF-8',
    'Referer':URL1,
    'User-Agent':'Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/48.0.2564.82 Chrome/48.0.2564.82 Safari/537.36',
    'Accept':'application/json, text/javascript,*/*'}
    
    with requests.Session() as s:
        #get html from URL1 in order to get the CSRF token
        page = s.get(URL1)
        CSRF=re.findall('WEATHERCLOUD_CSRF_TOKEN:"(.*)"},',page.text)[0]
        #create parameters for URL2, in order to get the json file
        params={'code':device,'WEATHERCLOUD_CSRF_TOKEN':CSRF}
        page_stats=requests.get(URL2,params=params,headers=headers)
        print(page_stats.url)
        print(page_stats) #<Response [200]>
        print(page_stats.text) #empty
        print(page_stats.json()) #error

但是页面_stats是空的。 如何从weathercloud获取统计数据

Answer 1

使用DevTools检查页面，您会发现一个有用的端点： https://app.weathercloud.net/device/stats

您可以使用requests库“复制”浏览器发出的原始web请求：

import requests

cookies = {
    'PHPSESSID': '************************',
   'WEATHERCLOUD_CSRF_TOKEN':'***********************',
    '_ga': '**********',
    '_gid': '**********',
    '__gads': 'ID=**********',
    'WeathercloudCookieAgreed': 'true',
    '_gat': '1',
    'WEATHERCLOUD_RECENT_ED3C8': '*****************',
}

headers = {
    'Connection': 'keep-alive',
    'sec-ch-ua': '^\\^Google',
    'Accept': 'application/json, text/javascript, */*; q=0.01',
    'X-Requested-With': 'XMLHttpRequest',
    'sec-ch-ua-mobile': '?0',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36',
    'sec-ch-ua-platform': '^\\^Windows^\\^',
    'Sec-Fetch-Site': 'same-origin',
    'Sec-Fetch-Mode': 'cors',
    'Sec-Fetch-Dest': 'empty',
    'Referer': 'https://app.weathercloud.net/d0838117883',
    'Accept-Language': 'it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7,es;q=0.6',
}

params = (
    ('code', '0838117883'),
    ('WEATHERCLOUD_CSRF_TOKEN', '****************'),
)

response = requests.get('https://app.weathercloud.net/device/stats', headers=headers, params=params, cookies=cookies)
# Serializing json
json_object = json.loads(response.text)

json输出：

{'last_update': 1632842172,
 'bar_current': [1632842172, 1006.2],
 'bar_day_max': [1632794772, 1013.4],
 'bar_day_min': [1632845772, 1006.2],
 'bar_month_max': [1632220572, 1028],
 'bar_month_min': [1632715572, 997.3],
 'bar_year_max': [1614418512, 1038.1],
 'bar_year_min': [1615434432, 988.1],
 'wdir_current': [1632842172, 180],
  ..............}

就这样