无法单击下一页进行web刮取

Question

我正试图从这个网站上搜集学校数据http://sekolah.data.kemdikbud.go.id/

点击“Cari Sekolah”按钮进入第一页进行刮除

我似乎无法找到一种方法来单击“下一页”以加载其他页面的条目。单击下一页时，url不会更改。不同页面的HTML代码如下所示：

<a href="javascript:void(0)" onclick="paging(2)">2</a>

这是我的密码：

from selenium import webdriver
from selenium.webdriver import Chrome
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd
import time

option = webdriver.ChromeOptions()
option.add_argument('--incognito')
webdriver = "/Users/rainer/Desktop/learnpython/web/chromedriver"
driver = Chrome(executable_path=webdriver, chrome_options=option)

url="http://sekolah.data.kemdikbud.go.id/"
driver.get(url)
WebDriverWait(driver,15).until(EC.element_to_be_clickable((By.XPATH,"//button[text()='Cari Sekolah']"))).click()
for i in range(2,15):
    try:
        WebDriverWait(driver, 15).until(EC.element_to_be_clickable((By.XPATH, "//ul[@id='list']//a[text()='" + str(i) + "']"))).click()
        print("page number #" + str(i))
        time.sleep(1)
    except:
        print("No such element")
        break

Answer 1

我执行了你的代码，它似乎工作正常。页面的URL不会更改，但页面上显示的学校实际上会更改

看一看，我添加了打印学校名称的代码：

url="http://sekolah.data.kemdikbud.go.id/"
driver.get(url)
wait = WebDriverWait(driver,15)

wait.until(EC.element_to_be_clickable((By.XPATH,"//button[text()='Cari Sekolah']"))).click()
for i in range(2, 15):
    try:
        wait.until(EC.element_to_be_clickable((By.XPATH, "//ul[@id='list']//a[text()='" + str(i) + "']"))).click()
        print("page number #" + str(i))

        # Print school names to make sure they change per page
        names = driver.find_elements(By.CSS_SELECTOR, 'ul > li.list-group-item:first-child a')
        print([name.text for name in names])
        time.sleep(2) 

    except Exception as e:
        print(e)
        break

输出：

page number #2
['(69839180) KELOMPOK BERMAIN KUSUMA HATI', '(K5660869) LKP ALVY', '(20103743) SMKS PGRI 1 JAKARTA', '(20269763) TK MARDI WALUYA']
page number #3
['(69990549) KB GENERASI CERDAS', '(69953341) KB HARAPAN BUNDA', '(P2965122) PKBM TELAGO', '(69930497) SPS ANGGREK']
page number #4
['(69877674) KB WADAH KASIH', '(20567521) TK AISYIYAH BUSTANUL ATHFAL', '(30406867) TK AISYIYAH BUSTANUL ATHFAL 2', '(69734039) TK PGRI TINGGAR']
...

因此，看起来您仍然可以刮取数据，并且一切都按预期工作

希望这能有所帮助。如果您的代码输出不同，请告诉我。祝你好运