网站垃圾处理表（无法找到表或属性）

Question

我正试图在http://tickertrak.com/上浏览表格，但我做不到。代码在table标记之后没有读取任何内容，我甚至看不到它，所以我非常困惑。我是网络垃圾新手，到目前为止只会做维基百科表格

import time
!pip install selenium
!apt install chromium-chromedriver
!cp /usr/lib/chromium-browser/chromedriver /usr/bin
import sys
sys.path.insert(0,'/usr/lib/chromium-browser/chromedriver')
import pandas as pd
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.headless = True
driver = webdriver.Chrome(options=options)

driver.get("http://tickertrak.com/")
time.sleep(2)

df = pd.read_html(driver.page_source, flavor="bs4")
df = pd.concat(df)
df.drop(index=0, axis=0, inplace=True)
df.to_csv("your_table.csv", index=False)`

我的线路出现错误：

driver = webdriver.Chrome(options=options)

用于webdriverexception。我忘了什么地方的路了吗

Answer 1

该表是由JS（JavaScript）动态生成的，因此您不会使用纯requests和bs4获取数据

但是，您可以给selenium一个快照并将其与panadas组合

以下是方法：

import time

import pandas as pd
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.headless = True
driver = webdriver.Chrome(options=options)

driver.get("http://tickertrak.com/")
time.sleep(2)

df = pd.read_html(driver.page_source, flavor="bs4")
df = pd.concat(df)
df.drop(index=0, axis=0, inplace=True)
df.to_csv("your_table.csv", index=False)

这将生成一个.csv文件，如下所示：

Answer 2

这是使用请求模块从该网页获取数据的更快方法之一，因为数据已经在脚本标记内的页面源中。您现在要做的就是在将数据存储到dataframe之前清理数据

import re
import requests

URL = 'http://tickertrak.com/'

with requests.Session() as s:
    s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36'
    r = s.get(URL)
    items = re.findall(r"var arrayFromPHP = \[(.*?)\];",r.text)[0]
    trs = re.findall(r"\[(.*?)\]",items)
    for tds in trs:
        print(tds)

输出如下：

"Options","gamestop corp","gme","1","58662","131","-80","-85","1"
"Options","amc entertainment holdings inc","AMC","1","16290","36","-79","-66","2"
"Options","nokia corp","nok","1","3568","14","-86","-88","3"
"Options","regal-beloit corp","RBC","1","3254","11","-56","-89","4"
"Options","blackberry ltd","BB","1","3002","10","-91","-92","5"