如何从任何网站上爬取表格并存储为数据框架？

Question

我需要从https://en.wikipedia.org/wiki/List_of_postal_codes_of_Canada:_M中删除一张桌子 并将这些数据存储在python数据帧中。 我拉过桌子，但无法选择列（邮政编码、自治区、邻里）

我的桌子是这样的：

<table class="wikitable sortable">
<tbody><tr>
<th>Postcode</th>
<th>Borough</th>
<th>Neighbourhood
</th></tr>
<tr>
<td>M1A</td>
<td>Not assigned</td>
<td>Not assigned
</td></tr>
<tr>
<td>M2A</td>
<td>Not assigned</td>
<td>Not assigned
</td></tr>
<tr>
<td>M3A</td>
<td><a href="/wiki/North_York" title="North York">North York</a></td>
<td><a href="/wiki/Parkwoods" title="Parkwoods">Parkwoods</a>
</td></tr>
<tr>
<td>M4A</td>
<td><a href="/wiki/North_York" title="North York">North York</a></td>
<td><a href="/wiki/Victoria_Village" title="Victoria Village">Victoria Village</a>
</td></tr>
...

url = 'https://en.wikipedia.org/wiki/List_of_postal_codes_of_Canada:_M'
response = requests.get(url)
soup= BeautifulSoup(response.text, "html.parser")

table = soup.find('table', {'class': 'wikitable sortable'})

df = []

for row in table.find_all('tr'):
    columns = row.find_all('td')
    Postcode = row.columns[1].get_text()
    Borough = row.columns[2].get_text()
    Neighbourhood = row.column[3].get_text()
    df.append([Postcode,Borough,Neighbourhood])

用上面的代码我得到 TypeError:“NoneType”对象不可订阅

我在谷歌上搜索了一下，才知道我做不到 邮编=行.列[1] .get\文本（） 因为函数的内联属性。你知道吗

我也尝试了其他方法，但得到了一些“索引错误消息”。你知道吗

很简单。我需要遍历该行，继续为每行选取三列，并将其存储在列表中。但我不能用代码来写。你知道吗

预期输出为

 Postcode   Borough   Neighbourhood
    M1A     Not assigned Not assigned
    M2A     Not assigned Not assigned
    M3A     North York    Parkwoods

Answer 1

我不知道熊猫，但我用这个脚本刮桌子。希望对你有帮助。你知道吗

import requests
from bs4 import BeautifulSoup
url = 'https://en.wikipedia.org/wiki/List_of_postal_codes_of_Canada:_M'
response = requests.get(url)
soup= BeautifulSoup(response.text, "html.parser")

tbl= soup.find('table', {'class': 'wikitable sortable'})
table_dict = {
    "head": [th.text.strip() for th in tbl.find_all('th')],
    "rows": [
        [td.text.strip() for td in tr.find_all("td")]
            for tr in tbl.find_all("tr")
                if not tr.find("th")
    ]
}

Answer 2

如果你想从网上刮一张桌子，你可以使用熊猫图书馆。你知道吗

import pandas as pd
url = 'valid_url'
df = pd.read_html(url)
print(df[0].head())

Answer 3

刮削的代码在下面的部分是错误的。你知道吗

import requests
from bs4 import BeautifulSoup
url = 'https://en.wikipedia.org/wiki/List_of_postal_codes_of_Canada:_M'
response = requests.get(url)
soup= BeautifulSoup(response.text, "html.parser")

table = soup.find('table', {'class': 'wikitable sortable'})

df = []

for row in table.find_all('tr'):
    columns = row.find_all('td') # the first row is returning <th> tags, but since you queried <td> tags, it's returning empty list.
    if len(columns)>0: #In order to skip first row or in general, empty rows, you need to put an if check.
        #Use the indices properly to get different values.
        Postcode = columns[0].get_text()
        Borough =columns[1].get_text()
        Neighbourhood = columns[2].get_text()
        df.append([Postcode,Borough,Neighbourhood])

再次，请注意，使用get\u text也会完整地返回链接和锚定标记。你可能想修改代码来避免这种情况。 开心网抓拍：）