如何访问通过javascript绑定呈现的内容？

Question

我已经试过把所有的球员数据从

https://www.easports.com/madden-nfl/player-ratings?i=1&s=ovr_rating:DESC&v=true&=undefined

用靓汤： 博客.py你知道吗

import requests
from bs4 import BeautifulSoup
from csv import writer

response = requests.get('https://www.easports.com/madden-nfl/player-ratings/?i=1&s=ovr_rating:DESC&v=true&=undefined')
soup = BeautifulSoup(response.text, 'html.parser')

posts = soup.find_all(class_='player_rating-value')
print(posts)

但它给了我一个空列表，或者如果我尝试使用

posts = soup.find_all(class_='ratings-hub_database')
print(posts)

它只会给我力量之前的数据

我读过其他的解决方案，上面说我需要从“网络”选项卡获取XHR数据，但我不知道如何获取

Answer 1

打开“网络”选项卡，然后按F5刷新页面。然后按Ctrl+F打开搜索框，输入玩家姓名，然后按enter。查看返回的结果，您将发现以下以json的形式返回统计信息

from bs4 import BeautifulSoup as bs
import requests

data = requests.get('https://www.easports.com/madden-nfl/ratings/service/data?entityType=madden19_player&filter=iteration:1&sort=ovr_rating:DESC,%20lastName:ASC&limit=25').json()
print(data)

---

您可以写入csv（请注意，您需要添加自己的头排序）

from bs4 import BeautifulSoup as bs
import requests
import pandas as pd

data = requests.get('https://www.easports.com/madden-nfl/ratings/service/data?entityType=madden19_player&filter=iteration:1&sort=ovr_rating:DESC,%20lastName:ASC&limit=25').json()
df = pd.DataFrame(data['docs'])
df.to_csv(r'C:\Users\User\Desktop\Info.csv', sep=',', encoding='utf-8-sig',index = False )

---

使用csv

from bs4 import BeautifulSoup as bs
import requests, csv

data = requests.get('https://www.easports.com/madden-nfl/ratings/service/data?entityType=madden19_player&filter=iteration:1&sort=ovr_rating:DESC,%20lastName:ASC&limit=25').json()
with open("data.csv", "w", encoding="utf-8-sig", newline='') as csv_file:
    w = csv.writer(csv_file, delimiter = ",", quoting=csv.QUOTE_MINIMAL)
    w.writerow(list(data['docs'][0].keys()))
    for row in data['docs']:
        w.writerow(list(row.values()))