点击两个按钮后的网页抓取

Question

我试图通过点击2个单选按钮，然后从如下列表中选择，从这个网站上获取数据http://rgphentableaux.hcp.ma/Default1/

我需要对该列表中的所有可用选项执行此操作，并将表添加到我拥有的数据帧中 这是我到目前为止尝试过的，但没有成功

    from bs4 import BeautifulSoup 
    pip install selenium
    from selenium import webdriver
    browser=webdriver.Chrome()
    url = "http://rgphentableaux.hcp.ma/Default1/"
    browser.get(url) #navigate to the page
    browser.find_element_by_xpath(".//input[@type='radio' and 
                     @value='5']").click()
    browser.find_element_by_id("CGEO").click()
    time.sleep(3)
    browser.find_element_by_xpath(".//input[@type='button' and 
                    @value='Afficher']").click()
    tabs = browser.find_elements_by_id('IEE')
    innerHTML = browser.execute_script("return 
                 document.body.innerHTML")
    soup_level2=BeautifulSoup(innerHTML, 'html.parser')

附言：我也需要弄到这里的桌子

Answer 1

您可以通过模仿页面发出的请求来使用requests和bs4完成整个过程。您只需按正确的顺序循环区域，并将当前区域编号添加到每个请求中的'CGEO'参数

---

这：

soup = bs(s.get(url).content, 'lxml')
regions = [i.text.strip() for i in soup.select('#REGIONSLIST option')]

从登录url收集区域名称的初始列表

---

这：

for k,v in regions.items():
    params = (('type', 'Region'), ('CGEO', v), ('them', '5'))

使用区域的optiontagvalue属性设置CGEO参数，例如。 Tanger-Tetouan-Al Hoceima是'01'

Region选项在type参数内设置

Langues locales utilisées选项在them参数内设置，即'5'

---

这：

for y in range(3):
    row.extend([data[i-y+2]['DATA2014']])

只需反转项的顺序，以便data内每个字典中的Ens, Fem, Masc以所需的输出顺序Masc, Fem, Ens添加到row

---

Py:

import requests
import pandas as pd
from bs4 import BeautifulSoup as bs

def add_rows(region, data):
    for i in range(0, len(data)//3, 3):
        row = [region, data[i]['INDICATEUR'].split('_')[-1]]
        for y in range(3):
            row.extend([data[i-y+2]['DATA2014']])
        final.append(row)
        
url = 'http://rgphentableaux.hcp.ma/Default1'    
headers= {'User-Agent': 'Mozilla/5.0',  'Referer': url}    
final = []

with requests.Session() as s:
    s.headers = headers
    soup = bs(s.get(url).content, 'lxml')
    regions = {i.text.strip():i['value'].strip() for i in soup.select('#REGIONSLIST option')}
    
    for k,v in regions.items():
        params = (('type', 'Region'), ('CGEO', v), ('them', '5'))
        r = s.get(f'{url}/getDATA/', params=params)
        data = r.json()
        add_rows(k, data)
    
df = pd.DataFrame(final, columns = ['Region', 'Lang', 'Masc', 'Fem', 'Ens'])
print(df)

---

编辑：

要获取所有3个表（ensemble、urbain、rural），请按如下所示调整自定义函数，并添加到附加循环for n in range(0, len(data), block)：

import requests
import pandas as pd
from bs4 import BeautifulSoup as bs

def add_rows(table, region, data_block):
    for i in range(0, len(data_block), 3):
        row = [table, region, data_block[i]['INDICATEUR'].split('_')[-1]]
        for y in range(3):
            row.extend([data_block[i-y+2]['DATA2014']])
        final.append(row)
        
url = 'http://rgphentableaux.hcp.ma/Default1'    
headers= {'User-Agent': 'Mozilla/5.0',  'Referer': url}
tables = ['ens', 'urb', 'rur']
final = []

with requests.Session() as s:
    s.headers = headers
    soup = bs(s.get(url).content, 'lxml')
    regions = {i.text.strip():i['value'].strip() for i in soup.select('#REGIONSLIST option')}
    
    for k,v in regions.items():
        params = (('type', 'Region'), ('CGEO', v), ('them', '5'))
        r = s.get(f'{url}/getDATA/', params=params)
        data = r.json()
        block = len(data)//3
        
        for n in range(0, len(data), block):
            table = tables[n//block]
            add_rows(table, k, data[n:n+block])
            
df = pd.DataFrame(final, columns = ['Table', 'Region', 'Language', 'Masc', 'Fem', 'Ens'])
print(df)