如何从1mg webaite中刮取img标签

Question

我正试图从1mg网站上刮取网页的HTML

在URL中，当我尝试将其保存为HTML时，或者当我尝试使用BeautifulSoup将其刮取时，返回None

用于刮削的代码：

from bs4 import BeautifulSoup
import requests
headers = {"User-Agent":'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.116 Safari/537.36'}
url='https://www.1mg.com/categories/ayurveda/top-brands-265?filter=true&brand=Dabur'
page= requests.get(url, headers=headers)
soup=BeautifulSoup(page.content,'html.parser')
img=soup.find_all('img',{'class':'style__image___Ny-Sa style__loaded___22epL'})
for i in img:
 i['src']

这是要刮取的图像的示例标记：

 <img alt="Dabur Shilajit Gold Capsule" src="https://res.cloudinary.com/du8msdgbj/images/w_150,h_150,c_fit,q_auto,f_auto/v1603435745/feaoalhp4c6bv8icllgp/dabur-shilajit-gold-capsule.jpg" title="Dabur Shilajit Gold Capsule" class="style__image___Ny-Sa style__loaded___22epL">

我已经手动复制了上面的标记作为示例。我对产品名称和价格使用了相同的代码（更改了标签），效果很好。我甚至尝试使用img标签的父标签

Answer 1

页面是动态加载的，因此requests不支持它。但是，该数据在网站上以JSON格式提供，请尝试使用内置的^{}模块获取所有图像（总共40个）

import json
import requests
from bs4 import BeautifulSoup

URL = "https://www.1mg.com/categories/ayurveda/top-brands-265?filter=true&brand=Dabur"

soup = BeautifulSoup(requests.get(URL).content, "html.parser")
json_data = json.loads(
    soup.select_one("#content-container > div > div > div > script:nth-child(5)").string
)

for data in json_data["itemListElement"]:
    print(data["image"])

输出：

https://res.cloudinary.com/du8msdgbj/images/w_150,h_150,c_fit,q_auto,f_auto/v1603435745/feaoalhp4c6bv8icllgp/dabur-shilajit-gold-capsule.jpg
https://res.cloudinary.com/du8msdgbj/images/w_150,h_150,c_fit,q_auto,f_auto/v1500611141/wwkoja9giavml3tgyza4/dabur-musli-pak-laghu.jpg
..All the way until

https://res.cloudinary.com/du8msdgbj/images/w_150,h_150,c_fit,q_auto,f_auto/v1601446598/sujrsvjyzcuvpfwtekhz/anti-oxidants-combo-of-organic-india-tulsi-ginger-turmeric-25-tea-bag-and-dabur-honey-squeezy-225gm-buy-1-get-1-free.png