Python在iframe中请求html表

Question

如何使用请求来刮取此链接中的表？我试图使用请求，但由于表位于iframe中，html返回的结果不完整。 我只需要表格中的html，一旦我有了它，我想我可以用beatuifulsoup处理这个问题。 下面是我正在使用的编码：

url = 'https://www.rad.cvm.gov.br/ENETCONSULTA/frmGerenciaPaginaFRE.aspx?NumeroSequencialDocumento=89180&CodigoTipoInstituicao=2'
resp = requests.get(url, verify=False)

Answer 1

如果不想使用selenium，可以使用此脚本加载带有requests的表：

import re
import requests
from bs4 import BeautifulSoup

base_url = 'http://www.rad.cvm.gov.br/ENETCONSULTA/frmGerenciaPaginaFRE.aspx?NumeroSequencialDocumento=89180&CodigoTipoInstituicao=2'

# https://stackoverflow.com/questions/38015537/python-requests-exceptions-sslerror-dh-key-too-small
requests.packages.urllib3.disable_warnings()
requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS += ':HIGH:!DH:!aNULL'
try:
    requests.packages.urllib3.contrib.pyopenssl.util.ssl_.DEFAULT_CIPHERS += ':HIGH:!DH:!aNULL'
except AttributeError:
    # no pyopenssl support used / needed / available
    pass

with requests.session() as s:
    html_data = s.get(base_url, verify=False).text
    url = 'http://www.rad.cvm.gov.br/ENETCONSULTA/' + re.search(r"window\.frames\[0\]\.location='(.*?)'", html_data).group(1)
    soup = BeautifulSoup(s.get(url, verify=False).content, 'html.parser')

    print(soup.table.prettify())

印刷品：

<table id="ctl00_cphPopUp_tbDados">
 <tr>
  <td style="padding:8px 5px 8px 5px; background:#cccfd1; border-bottom:1px solid #fff !important; text-align:center; color:#ffffff; font:normal normal bold 12px 'Trebuchet MS', sans-serif;">
   Conta
  </td>
  <td style="padding:8px 5px 8px 5px; background:#cccfd1; border-bottom:1px solid #fff !important; text-align:center; color:#ffffff; font:normal normal bold 12px 'Trebuchet MS', sans-serif;">
   Descrição
  </td>
  <td style="padding:8px 5px 8px 5px; background:#cccfd1; border-bottom:1px solid #fff !important; text-align:center; color:#ffffff; font:normal normal bold 12px 'Trebuchet MS', sans-serif;">
   01/07/2019
   <br/>
   a
   <br/>
   30/09/2019
  </td>

... and so on.

Answer 2

实现这一点的最佳方法是使用Selenium，等待几秒钟，直到iframe加载，然后捕获iframe的内容

下面是一个如何做到这一点的示例：

import sys
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from time import sleep

url = 'https://www.rad.cvm.gov.br/ENETCONSULTA/frmGerenciaPaginaFRE.aspx?NumeroSequencialDocumento=89180&CodigoTipoInstituicao=2'
options = Options()
# activate the following two lines to run in headless mode.
# options.add_argument(' headless')
# options.add_argument(' disable-gpu')
options.add_argument("user-agent=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36")
# /usr/bin/chromedriver is the path where I've installed chromedriver.
driver = webdriver.Chrome('/usr/bin/chromedriver', chrome_options=options)
driver.get(url)
# Wait till iframe loads
sleep(5)
html = driver.execute_script("return document.getElementsByTagName('html')[0].innerHTML").encode('utf-8').strip()
# Now you have the fully-loaded HTML, you may continue to use getElementByTagName or a different library like bs4 to extract the content of the iframe. 
driver.close()