从下拉列表组合中提取所有链接

Question

我有一个示例网站，我想从网站中提取所有的“href链接”。它有两个下拉列表，一旦选择下拉列表，它将显示结果，并链接到手动下载。 它不会导航到不同的页面，而是在同一页面上显示结果。我已经提取了下拉列表的组合，我试图提取手动链接，但我找不到链接

代码如下

from selenium import webdriver
from selenium.webdriver.support.ui import Select
import time
from bs4 import BeautifulSoup
import requests


url = "https://www.cars.com/"

driver = webdriver.Chrome('C:/Users/webdrivers/chromedriver.exe')
driver.get(url)
time.sleep(4)

selectYear = Select(driver.find_element_by_id("odl-selected-year"))

data = []
for yearOption in selectYear.options:
    yearText = yearOption.text
    selectYear.select_by_visible_text(yearText)
    time.sleep(1)

    selectModel = Select(driver.find_element_by_id("odl-selected-model"))
    for modelOption in selectModel.options:
        modelText = modelOption.text
        selectModel.select_by_visible_text(modelText)
        data.append([yearText,modelText])

page = requests.get(url, headers=headers)
soup = BeautifulSoup(page.text, 'html.parser')

content = soup.findAll('div',attrs={"class":"odl-results-container"})

for i in content:
    x = i.findAll(['h3','span'])
    for y in x:
        print(y.get_text())

打印不显示任何数据。如何获取手册的链接？提前谢谢

Answer 1

您需要单击每个车型和年份的按钮，然后从Selenium webdriver中检索呈现的HTML页面源，而不是通过请求

在内部循环中添加以下内容：

button = driver.find_element_by_link_text("Select this vehicle")
button.click()

page = driver.page_source
soup = BeautifulSoup(page, 'html.parser')
content = soup.findAll('a',attrs={"class":"odl-download-link"})
for i in content:
    print(i["href"])

这将打印出：

http://www.fordservicecontent.com/Ford_Content/vdirsnet/OwnerManual/Home/Index?Variantid=6875&languageCode=EN&countryCode=USA&marketCode=US&bookcode=O91668&VIN=&userMarket=GBR
http://www.fordservicecontent.com/Ford_Content/vdirsnet/OwnerManual/Home/Index?Variantid=7126&languageCode=EN&countryCode=USA&marketCode=US&bookcode=O134871&VIN=&userMarket=GBR
http://www.fordservicecontent.com/Ford_Content/vdirsnet/OwnerManual/Home/Index?Variantid=7708&languageCode=EN&countryCode=USA&marketCode=US&bookcode=O177941&VIN=&userMarket=GBR
...