如何在python中使用BeautifulSoup刮取隐藏的表内容？

Question

我试图从一个股票网站上搜集数据，但问题是表的内容是隐藏的。该网站是http://www.moneycontrol.com/stocks/histstock.php

1.Select Index
2.Select S&P BSE MIDCAP
3.Filter data from Jan 2019 to Jan 2020 to get to the final page 
4.I want to scrape the table contents of this page

这就是我尝试用的汤

import requests
from bs4 import BeautifulSoup
link='http://www.moneycontrol.com/stocks/hist_index_result.php?indian_indices=25'
html=requests.get(link)
html.status_code #200
raw=html.content
soup=BeautifulSoup(raw,'html.parser') #have tried with xml and html5lib
soup.find_all('table',{'class':'tblchart'})
#output
[<table border="0" cellpadding="0" cellspacing="0" class="tblchart">
                    </table>]

我也尝试过使用硒，但结果是一样的

我很难获得信息

如有任何建议、回答或是朝着正确的方向轻推，我们将不胜感激

Answer 1

好了，伙计们，我实际上用硒解决了这个问题，我必须更新我的硒包，它就像一个符咒

我是这样做的：

  import pandas as pd
  from selenium import webdriver

  link='http://www.moneycontrol.com/stocks/histstock.php'

  driver=webdriver.Chrome()
  driver.get(link)

  #selecting the index in Step 1
  driver.find_element_by_xpath('//*[@id="wutabs2"]').click()

  #Selecting from the dropdown Index options in step 2
  drop=driver.find_element_by_xpath('//*[@id="indian_indices"]')
  drop.click()
  drop.send_keys('S&P BSE MIDCAP')      

  #select the month in step 3

  month=driver.find_element_by_xpath('/html/body/div[3]/div[3]/div/div[7]/div[2]/div[6]/table/tbody/tr/td[3]/form/div[2]/select[2]')
  month.click()
  month.send_keys('2019')

  #click on search 
  driver.find_element_by_xpath('/html/body/div[3]/div[3]/div/div[7]/div[2]/div[6]/table/tbody/tr/td[3]/form/div[4]/input[1]').click()

  #getting the contents
  for i in driver.find_elements_by_css_selector('table.tblchart'):
       a=i.text

  a=a.split('\n')

  #storing it as a data frame
  df=pd.DataFrame(a)

  #removing the first column as it contained table headers
  df.drop(df.iloc[0:1,:],inplace=True)

  #splitting the columns using space and storing them seperately
  df['Month']=df[0].str.split(' ', expand=True)[0]
  df['Year']=df[0].str.split(' ', expand=True)[1]
  df['Open']=df[0].str.split(' ', expand=True)[2]
  df['High']=df[0].str.split(' ', expand=True)[3]
  df['Low']=df[0].str.split(' ', expand=True)[4]
  df['Close']=df[0].str.split(' ', expand=True)[5]