无法从第三页开始分析名称

2024-10-02 18:15:15 发布

您现在位置：Python中文网/ 问答频道 /正文
8523 3

我用python创建了一个脚本，使用请求模块和BeautifulSoup库从网站获取不同成员的名称。脚本可以完美地从第一页和第二页获取名称。然而，它从第三页开始就删除了相同的名称。我可以注意到，下一页逻辑在__EVENTTARGET的值范围内，如dnn$ctr410$MemberSearch$grdMembers$ctl00$ctl02$ctl01$ctl07和dnn$ctr410$MemberSearch$grdMembers$ctl00$ctl02$ctl01$ctl09等等。脚本可以相应地增加数字，但第二页之后的结果保持不变

要填充此website的结果，只需单击搜索按钮而不做任何更改。然后，您可以单击2,3,4 e.t.c页面进入相关页面

我尝试过（从前两页中删除数据）：

import requests
from bs4 import BeautifulSoup

link = 'https://www.icsi.in/student/Members/MemberSearch.aspx?SkinSrc=%5BG%5DSkins/IcsiTheme/IcsiIn-Bare&ContainerSrc=%5BG%5DContainers/IcsiTheme/NoContainer'

with requests.Session() as s:
    s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1; ) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36'
    r = s.get(link)
    soup = BeautifulSoup(r.text,"lxml") 
    payload = {i['name']:i.get('value','') for i in soup.select('input[name]')}
    payload['__EVENTTARGET'] = 'dnn$ctr410$MemberSearch$btnSearch'

    page = 5
    while True:
        r = s.post(link,data=payload)
        soup = BeautifulSoup(r.text,"lxml")
        for item in soup.select("span[id$='_lblFullName']"):
            print(item.text)

        page+=2
        payload = {i['name']:i.get('value','') for i in soup.select('input[name]')}
        if len(str(page))==1:
            payload['__EVENTTARGET'] = 'dnn$ctr410$MemberSearch$grdMembers$ctl00$ctl02$ctl01$ctl0{}'.format(page)
        else:
            payload['__EVENTTARGET'] = 'dnn$ctr410$MemberSearch$grdMembers$ctl00$ctl02$ctl01$ctl{}'.format(page)

        payload['__dnnVariable'] = {'__scdoff':'1','__dnn_pageload':'__dnn_setScrollTop();'}
        payload['ScrollTop'] = '400'

How can I fetch the names from the rest of the pages after the second page?


Tags： thenameinpagepayloadsoupdnnbeautifulsoup
2条回答
网友
1楼 · 编辑于 2024-10-02 18:15:15

实际上，您需要包含完整的postPayload参数

我们必须在相同的Session内使用requests.Session()来实现这一点，因为网站分页使用的是基于__dnnVariable的旋转function，服务器在JS请求下接收它，该请求被转换为循环

实际上它的意思是Next

因此，我首先发出了一个GET请求，并获取了所需的params（其中一些是动态的，另一些是静态的）

然后，我在同一个session下发出了一个post请求

import requests
import re
from bs4 import BeautifulSoup
from urllib.parse import unquote

data = {
    '__EVENTTARGET': "dnn$ctr410$MemberSearch$btnSearch",
    '__EVENTARGUMENT': '',
    '__VIEWSTATEENCRYPTED': '',
    'dnn$ctlHeader$dnnSearch$Search': 'SiteRadioButton',
    'dnn$ctlHeader$dnnSearch$txtSearch': '',
    'dnn$ctr410$MemberSearch$txtFirstName': '',
    'dnn$ctr410$MemberSearch$txtLastName': '',
    'dnn$ctr410$MemberSearch$ddlMemberType': 0,
    'dnn$ctr410$MemberSearch$txtMembershipNumber': '',
    'dnn$ctr410$MemberSearch$txtCpNumber': '',
    'dnn$ctr410$MemberSearch$txtCity': '',
    'dnn$ctr410$MemberSearch$txtOrganisation': '',
    'dnn$ctr410$MemberSearch$txtAddress2': '',
    'dnn$ctr410$MemberSearch$txtAddress3': '',
    'dnn$ctr410$MemberSearch$txtEmail': '',
    'dnn_ctr410_MemberSearch_grdMembers_ClientState': '',
    'ScrollTop': 432,
    '__dnnVariable': '{"__scdoff":"1","__dnn_pageload":"__dnn_setScrollTop();"}'
}


def main(url):
    with requests.Session() as req:
        r = req.get(url)
        soup = BeautifulSoup(r.content, 'html.parser')
        data['StylesheetManager_TSSM'] = re.search(
            r"hf.value \+= '(.*?)\'", r.text).group(1)
        data['ScriptManager_TSM'] = unquote(soup.findAll('script', src=True)
                                            [2]['src']).split("=", 3)[-1]
        data['__VIEWSTATE'] = soup.find("input", id="__VIEWSTATE").get("value")
        data['__VIEWSTATEGENERATOR'] = soup.find(
            "input", id="__VIEWSTATEGENERATOR").get("value")
        data['__EVENTVALIDATION'] = soup.find(
            "input", id="__EVENTVALIDATION").get("value")

        for _ in range(10):
            r = req.post(url, data=data)
            soup = BeautifulSoup(r.content, 'html.parser')
            names = [name.text for name in soup.select("div.name_head")]
            page = soup.select_one(
                "a.rgCurrentPage").next_sibling['href'].split("'")[1]
            data['__EVENTTARGET'] = page
            data['__EVENTVALIDATION'] = soup.find(
                "input", id="__EVENTVALIDATION").get("value")
            data['__VIEWSTATE'] = soup.find(
                "input", id="__VIEWSTATE").get("value")
            print(names)


main("https://www.icsi.in/student/Members/MemberSearch.aspx")

Note: on each request you will get the data not in sort order since the search function is using random on the back-end server.

输出：

['SH. DILIP RAGHUNATH KOTWAL', 'SH. ARUNODAY ROY MUKHERJEE', 'SH. J SUBRAMANI', 'SH. R KRISHNAMANI', 'SH. R NARAYANASWAMI', 'SH. M V GOPALAKRISHNAN', 'SH. RAJAM KRISHNAMURTHY', 'SH. V SIVASUBRAMANIAN', 'SH. V RAGHAVENDRAN', 'SH. G V AIMAN']
['SH. K J MATHEW', 'SH. K K GHOSH', 'SH. SUBHASH CHANDER DHAWAN', 'SH. BABU RAM MAHESWARI', 'SH. S SWAMINATHAN', 'SH. T S A AIYER', 'SH. KOVILOOR VIJAYARAGHAVACHARI SAMPATHKUMAR', 'SH. M KRISHNAN', 'SH. R N BANSAL', 'SH. N V RAMAN']
['SH. R VENKATARAMANI', 'SH. UTPALENDU ROY CHOUDHURY', 'SH. LAKSHMI NARAYANAN V', 'SH. PARIJAT KUMAR HORE', 'SH. B R VENKATESAN', 'SH. KISHAN GOPAL SOMANI', 'SH. O P GANERIWALA', 'SH. P T KUPPUSWAMY', 'SH. U P MATHUR', 'SH. N N UPADHYAY']
['SH. N K BHANDARI', 'SH. S R C SETTY', 'SH. S V BALASUBRAMANIAN', 'SH. HOSHIE HIRJI MALGHAM', 'SH. KAIKOBAD SORABJI ITALIA', 'SH. K SIVADAS', 'SH. K K SIVARAMAKRISHNAN', 'SH. A CHANDRASEKARAN', 'SH. R PONNAMBALAM', 'SH. T K B VENKATARAMAN']
['SH. NARINDER PAL', 'SH. PARKASH ATAM', 'SH. K A PARTHASARATHY', 'SH. SURESH CHANDRA OSWAL', 'SH. MAHENDRA KANTILAL SHAH', 'SH. V. SANTHANAKRISHNA', 'SH. VASANT NARAYAN GOGATE', 'SH. MANEKLAL 
PATEL', 'SH. B N VISHWANATH', 'SH. B S L NARAYAN']
['SH. P L N VIJAYANAGAR', 'SH. SHREEPAD MARTAND  KORDE', 'SH. SHIV BHAGWAN KOTHARI', 'SH. R B POPLAI', 'SH. RAMESH KHANNA', 'SH. RAVINDER NATH JOSHI', 'SH. VIDYA SAGAR AGGARWAL', 'SH. ARVIND JAYKUMAR CHAKOTE', 'SH. V RAMASESHAN', 'SH. BADRINARAYAN BALDAWA']
['SH. C GOVINDANKUTTY', 'SH. A G MADHAVAN', 'SH. DHIRAJ NATH BHATTACHARYYA', 'SH. RAMESHWAR LAL INANI', 'SH. RAMESHWARDAS C DAGA', 'SH. R SUBRAMANIAN', 'SH. S M REGE', 'SH. NARENDRA KUMAR KAPOOR', 'SH. K RAMAMURTHI', 'SH. ROOPENDRA NARAYAN ROY']
['SH. KALYAN KUMAR MITRA', 'SH. KALYANASUNDARAM ', 'SH. N A SESHADRI', 'SH. RAJENDRA KUMAR JAIN', 'SH. BISWAJIT SEN', 'SH. RAMKRISHNA NATHOOMAL  AGRAWAL', 'SH. P C SHETH', 'SH. K S NATARAJAN', 
'SH. S N DAMLE', 'SH. A M FADIA']
['DR. K N M RAO', 'SH. IYER M. RAMASWAMY', 'SH. DILIP KANTI MAZUMDAR', 'SH. RAM CHANDRA NIGAM', 'SH. SUBRAHMANIAM VISWANATHAN', 'SH. SURESH KUMAR JERATH', 'SH. A Y SRINIVASAN', 'DR. S C GARG', 
'SH. CHANDRA PRAKASH SHARDA', 'SH. M P JAIN']
['SH. E S DWARKANATH', 'SH. MYSORE SHAMANNA  RAMACHANDRA', 'SH. SUBHASH CHANDER SINGHAL', 'SH. T T SINHA', 'SH. G R BHANDARI', 'SH. M P GOEL', 'SH. CHOKKANATHAPURAM SUBRAMANIAN  NATESAN', 'SH. 
V M PATEL', 'SH. BIJOY KUMAR AGARWALLA', 'SH. BAHADUR CHAND JAIN']
网友
2楼 · 编辑于 2024-10-02 18:15:15

如果我从有效负载中删除dnn$ctr410$MemberSearch$grdMembers$ctl00$ctl02$ctl01$ctl02等箭头按钮键，它就会开始工作

    name_length = len('dnn$ctr410$MemberSearch$grdMembers$ctl00$ctl02$ctl01$ctl02')

    for key in list(payload.keys()):
        if key.startswith('dnn') and len(key) == name_length:
            payload.pop(key)
            print(key)

但您可以使用来自αԋɱҽԃ αмєяιcαη答案的方法来确保只发送所需的值

import requests
from bs4 import BeautifulSoup

link = 'https://www.icsi.in/student/Members/MemberSearch.aspx?SkinSrc=%5BG%5DSkins/IcsiTheme/IcsiIn-Bare&ContainerSrc=%5BG%5DContainers/IcsiTheme/NoContainer'

with requests.Session() as s:
    s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1; ) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36'
    r = s.get(link)
    soup = BeautifulSoup(r.text,"lxml")

    payload = {i['name']:i.get('value','') for i in soup.select('input[name]')}
    payload['__EVENTTARGET'] = 'dnn$ctr410$MemberSearch$btnSearch'

    page = 5
    while True:

        r = s.post(link, data=payload)
        soup = BeautifulSoup(r.text, "lxml")
        for item in soup.select("span[id$='_lblFullName']"):
            print(item.text)

        page += 2

        payload = {i['name']:i.get('value','') for i in soup.select('input[name]')}
        payload['__EVENTTARGET'] = 'dnn$ctr410$MemberSearch$grdMembers$ctl00$ctl02$ctl01$ctl{:02}'.format(page)

        name_length = len('dnn$ctr410$MemberSearch$grdMembers$ctl00$ctl02$ctl01$ctl02')
        for key in list(payload.keys()):
            if key.startswith('dnn') and len(key) == name_length:
                payload.pop(key)
                print(key)

        payload['__dnnVariable'] = {'__scdoff':'1','__dnn_pageload':'__dnn_setScrollTop();'}
        payload['ScrollTop'] = '400'

编辑：页面使用更复杂的系统，10页后显示新链接，但带有旧值ctl07，ctl09。我使用带有箭头的“从按钮到下一页”的“名称”来代替此链接-开始时，它的值为ctrl28，但在10页之后，它的值为ctrl30（因为有更多的链接-它将链接...添加到下一页/上一页的10页列表中）

import requests
from bs4 import BeautifulSoup

link = 'https://www.icsi.in/student/Members/MemberSearch.aspx?SkinSrc=%5BG%5DSkins/IcsiTheme/IcsiIn-Bare&ContainerSrc=%5BG%5DContainers/IcsiTheme/NoContainer'

with requests.Session() as s:
    s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1; ) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36'
    r = s.get(link)
    soup = BeautifulSoup(r.text,"lxml")

    payload = {i['name']:i.get('value','') for i in soup.select('input[name]')}
    payload['__EVENTTARGET'] = 'dnn$ctr410$MemberSearch$btnSearch'

    page = 1  # I don't need it to generate lins, now I use it only to display page number
    while True:
        print('page:', page)
        page += 1

        r = s.post(link, data=payload)
        soup = BeautifulSoup(r.text, "lxml")
        for item in soup.select("span[id$='_lblFullName']"):
            print(item.text)

        payload = {i['name']:i.get('value','') for i in soup.select('input[name]')}

        name_length = len('dnn$ctr410$MemberSearch$grdMembers$ctl00$ctl02$ctl01$ctl28')
        for key in list(payload.keys()):
            if key.startswith('dnn') and len(key) == name_length:
                payload.pop(key)
                #print(key)

        # button with arrow to next page

        next_page = soup.select("input[class='rgPageNext']")
        if not next_page:
            break

        next_page = next_page[0]['name']
        print(next_page)
        payload[next_page] = ''

        payload['__dnnVariable'] = {'__scdoff':'1','__dnn_pageload':'__dnn_setScrollTop();'}
        payload['ScrollTop'] = '400'

相关问题 更多 >

    编程相关推荐