如何将一个包含多个页面和链接的网站整合成一个数据框架？

Question

我需要得到的数据从每个公司在下面的链接与所有内容内的链接。我需要每个公司的数据排成一行。我现在的问题是我不知道该怎么做。我不知道该采取哪种方法，从哪里开始

这是网站：https://www.adgm.com/public-registers/fsra

我试着把信息输入到我的代码中，至少试着从IDE打印出来，但是失败了，我不明白为什么

import requests
import pandas as pd
from bs4 import BeautifulSoup

res = requests.get("https://www.adgm.com/public-registers/fsra")
soup = BeautifulSoup(res.content,'html.parser')
table  = soup.find_all('.every-accord')

for element in table:
    print(element.text)

这是我一直在尝试的代码。每个表行都在一个类“everyaccord”中，我正试图得到它。它没有给我任何错误，但我也没有得到任何结果

谢谢你的帮助

Answer 1

您可以迭代容器：

import requests
from bs4 import BeautifulSoup as soup
d = soup(requests.get('https://www.adgm.com/public-registers/fsra').text, 'html.parser')
results = [[c.text for c in i.find_all('div', {'class':'col-sm-6'})]+[i.a['href'], i.find('div', {'class':'col-lg-5'}).text] for i in d.find_all('div', {'class':'every-accord'})]
no_headers = [[i for i in c[1:] if i not in {'Company Status', 'Address'}] for c in results]

输出：

[['160024', 'Active', 'Level 7, Al Sila Tower, Abu Dhabi Global Market Square, Al Maryah Island, Abu Dhabi, United Arab Emirates', '/public-registers/fsra/fsf/aarna-capital-limited', 'Aarna Capital Limited'], ['160007', 'Active', 'Unit 8, 6th floor Al Khatem Tower, Abu Dhabi Global Markets Square, Al Maryah Island Abu Dhabi, United Arab Emirates P.O. Box 764605', '/public-registers/fsra/fsf/aberdeen-asset-middle-east-limited', 'Aberdeen Asset Middle East Limited'], ['180041', 'Active', 'Floor 22, Al Sila Tower, Abu Dhabi Global Market Square, Al Maryah Island', '/public-registers/fsra/fsf/abu-dhabi-catalyst-partners-limited', 'Abu Dhabi Catalyst Partners Limited'], ['180021', 'Active', 'Unit 5, 6th Floor, Al Khatem Tower, Abu Dhabi Global Market Square, Al Maryah Island, Abu Dhabi, United Arab Emirates', '/public-registers/fsra/fsf/ad-global-investors-limited', 'AD Global Investors Limited'], ['180039', 'Active', '3419, 34th Floor, Al Maqam Tower, Abu Dhabi Global Market Square, Al Maryah Island, Abu Dhabi, United Arab Emirates', '/public-registers/fsra/fsf/ad-investment-management-limited', 'AD Investment Management Limited'], ['170036', 'Active', '10th Floor, Al Sila Tower, ADGM Square, Al Maryah Island', '/public-registers/fsra/fsf/adcb-asset-management-ltd', 'ADCB Asset Management Ltd.'], ['160006', 'Active', 'Level 34, Al Khatem Tower, Abu Dhabi Global Market Square, Al Maryah Island, Abu Dhabi, United Arab Emirates', '/public-registers/fsra/fsf/adcm-altus-investment-management-limited', 'ADCM Altus Investment Management Limited'], ['160005', 'Active', '33rd floor, Al Khatem Tower, Abu Dhabi Global Market Square, Al Maryah Island, Abu Dhabi, United Arab Emirates', '/public-registers/fsra/fsf/adcorp-ltd', 'ADCORP Ltd'], ['180024', 'Active', 'Unit 10, Level 6, Al Khatem Tower, Abu Dhabi Global Market Square, Al Maryah Island, Abu Dhabi, United Arab Emirates', '/public-registers/fsra/fsf/adnoc-reinsurance-limited', 'ADNOC Reinsurance Limited'], ['170025', 'Active', 'Office 712, Al Khatem Tower, Abu Dhabi Global Market Square, Al Maryah Island, Abu Dhabi, United Arab Emirates', '/public-registers/fsra/fsf/ads-investment-solutions-limited', 'ADS Investment Solutions Limited']]

编辑：格式化results中的列：

new_results = [{**{j[i]:j[i+1] for i in range(0, len(j), 2)}, **{'link':a, 'name':b}} for *j, a, b in results]

输出：

[{'Financial Services Permission Number': '160024', 'Company Status': 'Active', 'Address': 'Level 7, Al Sila Tower, Abu Dhabi Global Market Square, Al Maryah Island, Abu Dhabi, United Arab Emirates', 'link': '/public-registers/fsra/fsf/aarna-capital-limited', 'name': 'Aarna Capital Limited'}, {'Financial Services Permission Number': '160007', 'Company Status': 'Active', 'Address': 'Unit 8, 6th floor Al Khatem Tower, Abu Dhabi Global Markets Square, Al Maryah Island Abu Dhabi, United Arab Emirates P.O. Box 764605', 'link': '/public-registers/fsra/fsf/aberdeen-asset-middle-east-limited', 'name': 'Aberdeen Asset Middle East Limited'}, {'Financial Services Permission Number': '180041', 'Company Status': 'Active', 'Address': 'Floor 22, Al Sila Tower, Abu Dhabi Global Market Square, Al Maryah Island', 'link': '/public-registers/fsra/fsf/abu-dhabi-catalyst-partners-limited', 'name': 'Abu Dhabi Catalyst Partners Limited'}, {'Financial Services Permission Number': '180021', 'Company Status': 'Active', 'Address': 'Unit 5, 6th Floor, Al Khatem Tower, Abu Dhabi Global Market Square, Al Maryah Island, Abu Dhabi, United Arab Emirates', 'link': '/public-registers/fsra/fsf/ad-global-investors-limited', 'name': 'AD Global Investors Limited'}, {'Financial Services Permission Number': '180039', 'Company Status': 'Active', 'Address': '3419, 34th Floor, Al Maqam Tower, Abu Dhabi Global Market Square, Al Maryah Island, Abu Dhabi, United Arab Emirates', 'link': '/public-registers/fsra/fsf/ad-investment-management-limited', 'name': 'AD Investment Management Limited'}, {'Financial Services Permission Number': '170036', 'Company Status': 'Active', 'Address': '10th Floor, Al Sila Tower, ADGM Square, Al Maryah Island', 'link': '/public-registers/fsra/fsf/adcb-asset-management-ltd', 'name': 'ADCB Asset Management Ltd.'}, {'Financial Services Permission Number': '160006', 'Company Status': 'Active', 'Address': 'Level 34, Al Khatem Tower, Abu Dhabi Global Market Square, Al Maryah Island, Abu Dhabi, United Arab Emirates', 'link': '/public-registers/fsra/fsf/adcm-altus-investment-management-limited', 'name': 'ADCM Altus Investment Management Limited'}, {'Financial Services Permission Number': '160005', 'Company Status': 'Active', 'Address': '33rd floor, Al Khatem Tower, Abu Dhabi Global Market Square, Al Maryah Island, Abu Dhabi, United Arab Emirates', 'link': '/public-registers/fsra/fsf/adcorp-ltd', 'name': 'ADCORP Ltd'}, {'Financial Services Permission Number': '180024', 'Company Status': 'Active', 'Address': 'Unit 10, Level 6, Al Khatem Tower, Abu Dhabi Global Market Square, Al Maryah Island, Abu Dhabi, United Arab Emirates', 'link': '/public-registers/fsra/fsf/adnoc-reinsurance-limited', 'name': 'ADNOC Reinsurance Limited'}, {'Financial Services Permission Number': '170025', 'Company Status': 'Active', 'Address': 'Office 712, Al Khatem Tower, Abu Dhabi Global Market Square, Al Maryah Island, Abu Dhabi, United Arab Emirates', 'link': '/public-registers/fsra/fsf/ads-investment-solutions-limited', 'name': 'ADS Investment Solutions Limited'}]