如何用Python正确地从NVIDIA网站上抓取？

Question

我正在使用BeautifulSoup4和请求

from bs4 import BeautifulSoup
import requests

url_NVIDIAGEFORCE = f'https://shop.nvidia.com/de-de/geforce/store/gpu/?page=1&limit=9&locale=de-de&category=GPU&gpu=RTX%203090,RTX%203080%20Ti,RTX%203080,RTX%203070%20Ti,RTX%203070,RTX%203060%20Ti,RTX%203060&gpu_filter=RTX%203090~12,RTX%203080%20Ti~7,RTX%203080~16,RTX%203070%20Ti~3,RTX%203070~18,RTX%203060%20Ti~8,RTX%203060~2,RTX%202080%20SUPER~1,RTX%202080~0,RTX%202070%20SUPER~0,RTX%202070~0,RTX%202060~6,GTX%201660%20Ti~0,GTX%201660%20SUPER~9,GTX%201660~8,GTX%201650%20Ti~0,GTX%201650%20SUPER~3,GTX%201650~17'


page = requests.get(url_NVIDIAGEFORCE).text
soup = BeautifulSoup(page, "lxml")
match = soup.find('div', class_='product_detail_78')
print(match)

几秒钟后，我得到输出：

None

这个类的div肯定存在，我从网站上复制了它

Answer 1

数据通过Json从外部URL加载。您可以使用以下示例来解析它：

import json
import requests


url = "https://api.nvidia.partners/edge/product/search?page=1&limit=9&locale=de-de&category=GPU&gpu=RTX%203090,RTX%203080%20Ti,RTX%203080,RTX%203070%20Ti,RTX%203070,RTX%203060%20Ti,RTX%203060&gpu_filter=RTX%203090~12,RTX%203080%20Ti~7,RTX%203080~16,RTX%203070%20Ti~3,RTX%203070~18,RTX%203060%20Ti~8,RTX%203060~2,RTX%202080%20SUPER~1,RTX%202080~0,RTX%202070%20SUPER~0,RTX%202070~0,RTX%202060~6,GTX%201660%20Ti~0,GTX%201660%20SUPER~9,GTX%201660~8,GTX%201650%20Ti~0,GTX%201650%20SUPER~3,GTX%201650~17"
headers = {
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:89.0) Gecko/20100101 Firefox/89.0"
}

data = requests.get(url, headers=headers).json()

# uncomment to print all data:
# print(json.dumps(data, indent=4))

for p in data["searchedProducts"]["productDetails"]:
    print("{:<50} {}".format(p["productTitle"], p["productPrice"]))

印刷品：

NVIDIA GEFORCE RTX 3090                            €1,549.00
NVIDIA GEFORCE RTX 3070                            €519.00
NVIDIA GEFORCE RTX 3060 Ti                         €419.00
NVIDIA GEFORCE RTX 3070 Ti                         €619.00
NVIDIA GEFORCE RTX 3080 Ti                         €1,199.00
ZOTAC GAMING GeForce RTX 3060 Twin Edge OC         €772.25
ZOTAC GAMING GEFORCE RTX 3060 Ti Twin Edge         €893.20
EVGA GeForce RTX 3060 Ti XC Gaming                 €896.29
Gigabyte GeForce RTX 3060 Ti EAGLE OC 8G           €925.82