用Python中的搜索框和javascripts抓取特定网站

Question

在网站https://sray.arabesque.com/dashboard上有一个html格式的搜索框“输入”。我想在搜索框中输入一个公司名称，在下拉菜单中选择该名称的第一个建议（例如，“Anglo American plc”），转到包含该公司信息的url，加载javascripts以获得所获得页面的完整html版本，然后在底部刮取它的GC分数、ESG分数、温度分数

!apt install chromium-chromedriver
!cp /usr/lib/chromium-browser/chromedriver /usr/bin
!pip install selenium

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
options = webdriver.ChromeOptions()
options.add_argument('-headless')
options.add_argument('-no-sandbox')
options.add_argument('-disable-dev-shm-usage')

wd = webdriver.Chrome('chromedriver',options=options)

companies = ['Anglo American plc']

for company in companies:
  # dryscrape.start_xvfb()
  # session = dryscrape.Session()
  # session.visit("https://srayapi.arabesque.com/api/sray/company/history/004BTP-E")
  resp = wd.get('https://sray.arabesque.com/dashboard/')
#print(driver.page_source)
  e = wd.find_element_by_id(id_='mat-input-0')
  e.send_keys(company)
  e.send_keys(Keys.ENTER)
  innerHTML = e.execute_script("return document.body.innerHTML")
  print(innerHTML)

我不太明白如何访问包含英美资源集团信息的URL，如果在搜索框中输入公司名称后我们不知道该URL，如何将其删除

Answer 1

您可以使用selenium实现这一点。您需要更新一些内容

在与headless交互时，您需要提供window size

诱导WebDriverWait（）以避免同步问题

代码：

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

options = webdriver.ChromeOptions()
options.add_argument('-headless')
options.add_argument('-no-sandbox')
options.add_argument('-disable-dev-shm-usage')
options.add_argument('window-size=1920,1080')

wd = webdriver.Chrome(options=options)

companies = ['Anglo American plc']

for company in companies:
  wd.get('https://sray.arabesque.com/dashboard/')
  WebDriverWait(wd, 20).until(EC.element_to_be_clickable((By.XPATH, "//a[text()='list']"))).click()
  WebDriverWait(wd, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@id='mat-input-0']"))).send_keys(company)
  WebDriverWait(wd, 20).until(EC.element_to_be_clickable((By.XPATH, "//span[contains(.,' Anglo American plc ')]"))).click()
  WebDriverWait(wd, 20).until(EC.element_to_be_clickable((By.XPATH, "(//span[normalize-space(.)='Open dashboard'])[1]"))).click()
  WebDriverWait(wd,10).until(EC.visibility_of_element_located((By.CSS_SELECTOR,"div.mat-tab-labels")))
  print(wd.find_element_by_xpath("//div[@class='mat-tab-label-content'][contains(.,'GC Score')]/span").text)
  print(wd.find_element_by_xpath("//div[@class='mat-tab-label-content'][contains(.,'ESG Score')]/span").text)
  print(wd.find_element_by_xpath("//div[@class='mat-tab-label-content'][contains(.,'Temp')]/span").text)

输出：

57.03
53.78
2.7°C


 

Answer 2

在不确切了解您为什么要使用selenium的情况下，使用搜索，然后获取另一个站点，下面是我将如何获取您要查找的数据：

import requests
import json

session = requests.Session()
url = 'https://srayapi.arabesque.com/api/sray/q'
response = session.get(url).json()

rays = response['data']['rays']
[ray for ray in rays if ray['name'].startswith('Anglo American')]

然后做你想做的任何事情，因此对于esg、gc和温度可能：

myObj = [{result['name']: {'gc': result['gc'], 'esg': result['esg'], 'temp': result['score_near']}} for result in results]