Python、web抓取和BeautifulSoup但不起作用

Question

我想从下面的html代码中获取所有产品的列表，以及它们是“instock”还是“outofstock”

<div class="js-product-sizer sizes__layout built" data-scope="conversion-zone">   
            <div class="sizes__wrapper sizes__wrapper--visible" data-id="1042303" data-sellableonline="true">
              <span class="sizes__button sizes__button--selected" role="button" data-quantity="">Taglia</span>
              <ul class="sizes__list" role="listbox">
                  <li class="sizes__size" data-id="969834" data-belowthreshold="false" data-quantity="true" data-available-quantity="1631" data-backinstockqualified="false" data-locale="it" data-storeprice="€1,49" data-storename="Cavallino, Lecce" data-price="1.49" data-weight="0.528" data-favstore-stock="10" data-favstore-above-threshold="false" data-favstore-cnc1h="true" aria-labelledby="size-selector-title" role="option">
                    <span class="sizes__info" data-tnr-size-selector-bootstrap-by-text="">0,5 KG</span>
                    <span class="sizes__stock"                                                             
                                  <span class="sizes__stock__info" data-tnr-size-selector-stock-info="">Disponibile</span>                        
                    </span>
                  </li>
                
                  <li class="sizes__size" data-id="969842" data-belowthreshold="false" data-quantity="false" data-available-quantity="0" data-backinstockqualified="true" data-locale="it" data-storeprice="€3,49" data-storename="Cavallino, Lecce" data-price="3.49" data-weight="1.074" data-displayname="Disco ghisa bodybuilding 28mm" data-favstore-stock="0" data-favstore-above-threshold="false" data-favstore-cnc1h="false" aria-labelledby="size-selector-title" role="option">
                    <span class="sizes__info" data-tnr-size-selector-bootstrap-by-text="">1 KG</span>
                    <span class="sizes__stock">   
                              <span class="sizes__stock__info sizes__stock__info--nostock" data-tnr-size-selector-stock-info="">0 disponibili</span>
                            

我已运行以下代码：

import requests
from bs4 import BeautifulSoup
import time

r = requests.get('https://www.decathlon.it/p/disco-ghisa-bodybuilding-28mm/_/R-p-7278?mc=1042303&c=NERO')
soup = BeautifulSoup(r.text, 'html.parser')

for anchor_tag in soup.find_all(class_="js-product-sizer sizes__layout built")[0].findChildren():
    if "sizes_stock" in anchor_tag['class']:
        print(f"Size {anchor_tag.text} OOS")
    else:
        print(f"Size {anchor_tag.text}  in stock!")

但它给了我以下错误：

IndexError: list index out of range

Answer 1

您可以通过调用

r.raise_for_status()

或手动检查r.status_code

您将看到requests.exceptions.HTTPError: 403 Client Error: Forbidden for url:...。因此，在您的响应文本中确实没有该类的div，并且获取索引为0的元素是不可能的，并且会引发IndexError。你试图抓取的网站正在检测简单的抓取请求，请尝试至少设置一些标题（也许fake_useragent可以帮助你）

Answer 2

如前所述，他们有一种方法来检测提取信息的自动化过程。如果您使用selenium访问页面a，它会显示html。因此，您可以使用selenium获取页面源代码，然后对其进行解析

也稍微理清了你的逻辑。您正在打印'in stock'，无论是否有任何可用数量

另外，如果在使用find_all()时只需要拉取第一个元素（soup.find_all(class_="js-product-sizer sizes__layout built")[0]），那么最好只使用find()，因为它返回的正是它找到的第一个元素

代码：

from selenium import webdriver
from bs4 import BeautifulSoup
import time

url = 'https://www.decathlon.it/p/disco-ghisa-bodybuilding-28mm/_/R-p-7278?mc=1042303&c=NERO'


while True:

    driver = webdriver.Chrome(executable_path=r'C:/chromedriver_win32/chromedriver.exe')

    meta = 'ROBOTS'
    while meta == 'ROBOTS':
        driver.get(url)
        page = driver.page_source 
        soup = BeautifulSoup(page,'html.parser')
        if 'name' in soup.find('meta').attrs:
            meta = soup.find('meta')['name']
        else:
            meta = False
        
    driver.close()

    anchor_tag = soup.find(class_="js-product-sizer sizes__layout built")
    items_list = anchor_tag.find_all('li')
    for item in items_list:
        if item['data-weight'] == '0.528':
            continue
        quantity = item['data-available-quantity']
        itemStr = item.text.split('\n')
        if int(quantity) > 0:
            itemStr.insert(2,quantity)
        itemStr = ' '.join(itemStr).strip()
        if item['data-available-quantity'] == '0':
            print(f"Size {itemStr} OOS")
        else:
            print(f"Size {itemStr}  in stock!")
    
    time.sleep(60)

输出：

Size 0,5 KG 1584  Disponibile  in stock!
Size 1 KG  0 disponibili OOS
Size 2 KG  0 disponibili OOS
Size 5 KG  0 disponibili OOS
Size 10 KG  0 disponibili OOS
Size 20 KG  0 disponibili OOS