无法使用selenium、beautifulsoup和python刮取卡的详细信息

Question

实际问题：

指向scrape的链接：https://www.axisbank.com/retail/cards/credit-card/axis-bank-ace-credit-card/features-benefits#menuTab

我希望从该链接获得的项目在以下图像中：

我编写了以下代码：

from urllib.request import urlopen
from bs4 import BeautifulSoup
import json, requests, re
from selenium import webdriver

s = 'https://www.axisbank.com/retail/cards/credit-card/axis-bank-ace-credit-card/features-benefits#menuTab'
driver = webdriver.Chrome(executable_path="C:\\Users\\Hari\\Downloads\\chromedriver.exe")
driver.get(s)
soup = BeautifulSoup(driver.page_source, 'lxml')
# print(x.find('h3').get_text())
det = []
a = soup.find('div', class_ = 'owl-stage')
for x in a.find_all('div', class_ = 'owl-item'):
    print(x.find('li').get_text())
driver.close()

我尝试了上面的代码，但在得到这个输出后被卡住了

输出

Traceback (most recent call last):
  File "C:\Users\Hari\PycharmProjects\Card_Prj\buffer.py", line 22, in <module>
    print(x.find('li').get_text())
AttributeError: 'NoneType' object has no attribute 'get_text'

我不知道如何进一步处理和获取我想要的信息，非常感谢您的帮助

Answer 1

要使用Selenium和python提取功能和好处部分中的可见文本，您必须为visibility_of_all_elements_located()诱导WebDriverWait，并且可以使用以下任一Locator Strategies：

• 使用CSS_SELECTOR：

  driver.get("https://www.axisbank.com/retail/cards/credit-card/axis-bank-ace-credit-card/features-benefits#menuTab")
  print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "div.owl-item.active div.contentBox")))])
  

• 使用XPATH：

  driver.get("https://www.axisbank.com/retail/cards/credit-card/axis-bank-ace-credit-card/features-benefits#menuTab")
  print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//div[@class='owl-item active']//div[@class='contentBox']")))])
  

• 控制台输出：

  ['Launch offer\n5% cashback on Big Basket and Grofers\nValid till 28th February 2021\nFor detailed terms and conditions, click here', 'Unlimited Cashback on every spend\n5% cashback on bill payments (electricity, internet, gas and more) DTH and mobile recharges on Google Pay\n4% on Swiggy, Zomato & Ola\n2% on all other spends\nNo upper limit on cashback\n\nRead More', 'Lounge Access\nEnjoy 4 complimentary lounge visits per calendar year at select domestic airports with your ACE Credit Card. For list of airports and detailed terms and conditions, click here']
  

• 注意：您必须添加以下导入：

  from selenium.webdriver.support.ui import WebDriverWait
  from selenium.webdriver.common.by import By
  from selenium.webdriver.support import expected_conditions as EC
  

Answer 2

编辑

如评论中所述，预期输出是另一个输出，应添加到问题中。无论如何，要实现您的目标，请提取如下标题和描述：

for x in soup.select('div.owl-stage div.owl-item'):
    heading = x.h3.get_text(strip=True)
    description = x.select_one('h3 + div').get_text(strip=True)
    det.append(heading+':'+description)