使用BeautifulSoup查找Xbox Game Pass游戏时丢失数据

Question

此page包含Xbox Game Pass上所有可用游戏的列表。我想使用BeautifulSoup检索游戏名称列表

这就是我正在做的：

import requests
from bs4 import BeautifulSoup

page = requests.get('https://www.xbox.com/en-US/xbox-game-pass/games?=pcgames')

if (page.status_code != 200):
    print("Unable to load game pass games page")
    exit()

soup = BeautifulSoup(page.content, 'html.parser')
soup.find_all('h3', class_='gameDevLink') # returns []

s = soup.prettify()

with open('dump.html', 'w', encoding='utf-8') as f:
    f.write(s)

如果我在页面上查看游戏，我会看到如下内容：

Inspecting a game's html

每个游戏都包含在一个“a”标记中，其class设置为gameDevLink

我的问题是soup.find_all('a', class_='gameDevLink')没有返回命中

如果我将BeautifulSoup生成的html保存到磁盘并搜索gameDevLink，那么同样没有命中

我不明白为什么我可以在浏览器中看到这些信息，但BeautifulSoup似乎看不到

Answer 1

有关游戏的信息通过Javascript从其他URL加载。您可以使用此脚本对其进行模拟：

import json
import requests
from bs4 import BeautifulSoup


headers = {'User-Agent': 'User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36'}
game_ids_url = 'https://catalog.gamepass.com/sigls/v2?id=fdd9e2a7-0fee-49f6-ad69-4354098401ff&language=en-us&market=US'
game_info_url = 'https://displaycatalog.mp.microsoft.com/v7.0/products?bigIds={}&market=US&languages=en-us&MS-CV=XXX'

game_ids = requests.get(game_ids_url).json()
s = ','.join(i['id'] for i in game_ids if 'id' in i)
    
data = requests.get(game_info_url.format(s)).json()

# uncomment this to print all data:
#print(json.dumps(data, indent=4))

# print some data to screen:
for p in data['Products']:
    print(p['LocalizedProperties'][0]['ProductTitle'])
    print(p['LocalizedProperties'][0]['ShortDescription'])
    print('-' * 80)

印刷品：

A Plague Tale: Innocence - Windows 10
Follow the grim tale of young Amicia and her little brother Hugo, in a heartrending journey through the darkest hours of history. Hunted by Inquisition soldiers and surrounded by unstoppable swarms of rats, Amicia and Hugo will come to know and trust each other. As they struggle to survive against overwhelming odds, they will fight to find purpose in this brutal, unforgiving world.
                                        
Age of Empires Definitive Edition
Age of Empires, the pivotal real-time strategy game that launched a 20-year legacy returns with modernized gameplay, all-new 4K visuals, 8-person multiplayer battles and a host of other new features.  Welcome back to history.
                                        
Age of Empires II: Definitive Edition
Age of Empires II: Definitive Edition celebrates the 20th anniversary of one of the most popular strategy games ever with stunning 4K Ultra HD graphics, a new and fully remastered soundtrack, and brand-new content, “The Last Khans” with 3 new campaigns and 4 new civilizations.

Choose your path to greatness with this definitive remaster to one of the most beloved strategy games of all time.
                                        
Age of Empires III: Definitive Edition
Age of Empires III: Definitive Edition completes the celebration of one of the most beloved real-time strategy franchises with remastered graphics and music, all previously released expansions and brand-new content to enjoy for the very first time. 
                                        
Age of Wonders: Planetfall
Emerge from the cosmic dark age of a fallen galactic empire to build a new future for your people. Age of Wonders: Planetfall is the new strategy game from Triumph Studios, creators of the critically acclaimed Age of Wonders series, bringing all the exciting tactical turn-based combat and in-depth empire building of its predecessors to space in an all-new, sci-fi setting.
                                        

...and so on (Total 204 games)