用python抓取.aspx页面

Question

我是新的网页抓取游戏。我正在尝试取消以下网站： http://www.foodemissions.com/foodemissions/Calculator.aspx

使用互联网上的资源，我组合了以下HTTP POST请求：

import urllib
from bs4 import BeautifulSoup

headers = {
    'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.17 (KHTML, like Gecko)  Chrome/24.0.1312.57 Safari/537.17',
    'Content-Type': 'application/x-www-form-urlencoded',
    'Accept-Encoding': 'gzip,deflate,sdch',
    'Accept-Language': 'en-US,en;q=0.8',
    'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3'
}

class MyOpener(urllib.FancyURLopener):
    version = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.57 Safari/537.17'

myopener = MyOpener()
url = 'http://www.foodemissions.com/foodemissions/Calculator.aspx'
# first HTTP request without form data
f = myopener.open(url)
soup_dummy = BeautifulSoup(f,"html5lib")
# parse and retrieve two vital form values
viewstate = soup_dummy.select("#__VIEWSTATE")[0]['value']
viewstategen = soup_dummy.select("#__VIEWSTATEGENERATOR")[0]['value']

soup_dummy.find(id="ctl00_MainContent_category")

#search for the string 'input' to find the form data
formData = (
    ('__VIEWSTATE', viewstate),
    ('__VIEWSTATEGENERATOR', viewstategen),
    ('ctl00$MainContent$transport', '200'),
    ('ctl00$MainContent$quantity','1'),
    ('ctl00$MainContent$wastepct','100')
)

encodedFields = urllib.urlencode(formData)
# second HTTP request with form data
f = myopener.open(url, encodedFields)
soup = BeautifulSoup(f,"html5lib")
trans_emissions = soup.find("span", id="ctl00_MainContent_transEmissions")
print(trans_emissions.text)

即使更改了ctl00$MainContent$transport元素，最后一个print命令的输出似乎也没有改变。有什么线索可以解释为什么会这样吗？在

谢谢！在

Answer 1

你需要让ASP.NETApp“think”认为您通过将按钮名称添加到__EVENTTARGET隐藏输入中，单击了计算按钮。在

formData = (
    ('__VIEWSTATE', viewstate),
    ('__VIEWSTATEGENERATOR', viewstategen),
    ('ctl00$MainContent$transport', '100'),
    ('ctl00$MainContent$quantity','150'),
    ('ctl00$MainContent$wastepct','200'),
    ('__EVENTTARGET', 'ctl00$MainContent$calculate')
)