使用Python从没有表单标签但有文本输入的网站抓取数据

first = urllib2.Request(url, data=urllib.urlencode({'value': CODE}), headers={'User-Agent' : 'Google Chrome' 'Cookie': 'name = ctl00$ContentPlaceHolder1$C001$tbUserAddress'})

post_params = { 'ctl00$ContentPlaceHolder1$C001$tbUserAddress': CODE } first = urllib.urlencode(post_params) driver = webdriver.Chrome() driver.get(url) sbox = driver.find_element_by_class_name("ctl00$ContentPlaceHolder1$C001$tbUserAddress") sbox.send_keys(CODE) driver.find_element_by_class_name("ctl00$ContentPlaceHolder1$C001$btnSearchClient").click()

br = mechanize.Browser() br.open(url) br.select_form(name='ctl00$ContentPlaceHolder1$C001$tbUserAddress') br['value'] = CODE br.submit() http = urllib2.urlopen(br.response()) soup = BeautifulSoup(http, "html5lib")

2条回答

网友

1楼 · 编辑于 2024-09-30 02:19:50

如果我正确理解您的问题，您需要发送请求，带有特定的参数，并检查响应。好的，让我们看看提交后发送到哪里的请求。让我们打开邮递员。Post request params

正如我们看到的ctl00$ContentPlaceHolder1$C001$tbUserAddress获取值100，以及ctl00$ContentPlaceHolder1$T6B6681F0008$DDRADIUS，ctl00$ContentPlaceHolder1$C001$ddlRadius，ctl00$cphTopBar$T917BC451013$rblRadius得到半径值25。在

所以让我们获取一个包含数据的小片段来发送post请求并获得所需的响应

我使用python请求

和lxml来解析html响应

我更喜欢lxml，它比beauthulsoup更难理解，但要快得多。在

import requests
from lxml import html

input_data = {
    'ctl00$cphTopBar$T917BC451013$rblRadius': 25,
    'ctl00$ContentPlaceHolder1$T6B6681F0008$ddlRadius': 25,
    'ctl00$ContentPlaceHolder1$C001$ddlRadius': 25,
    'ctl00$ContentPlaceHolder1$C001$tbUserAddress': 100
}
resp = requests.post('http://www.sleepeducation.org/find-a-facility', data=input_data)
tree = html.fromstring(resp.text)
print(tree.xpath('//div[@id="ContentPlaceHolder1_C001_map_canvas"]')[0])

我没有足够的声誉来放置文档链接，我会尝试将它们放在评论中，或者您可以通过googlepythonrequests和pythonlxml 你也可以用美体素来做

^{pr2}$

网友

2楼 · 编辑于 2024-09-30 02:19:50

这对我有用

from bs4 import BeautifulSoup
from selenium import webdriver

url = 'http://www.sleepeducation.org/find-a-facility'
subButton = 'ContentPlaceHolder1_C001_btnSearchClient'
addyName = 'ctl00$ContentPlaceHolder1$C001$tbUserAddress'
addyId = 'ContentPlaceHolder1_C001_tbUserAddress'

def usingChromeSelenium():
    driver = webdriver.Chrome('C:\Users\documents\chromedriver.exe')
    driver.get(url)
    sleep(1)
    driver.find_element_by_name(addyName).send_keys(CODE)
    driver.find_element_by_id(subButton).click()
    sleep(1)
    html = driver.page_source
    return html

results = usingChromeSelenium()
soup = BeautifulSoup(results, "html.parser")

对于“网络驱动程序.Chrome（）“您必须下载chrome.exe应用程序文件，并在括号中包含文件的路径，如果没有路径，它可能对您有效

相关问题更多 >

编程相关推荐

热门问题

热门文章