使用webscraping从网站下载多个pdf

Question

大家好，我需要一些关于我的网络刮板的帮助，因为我想从https://jbiomedsci.biomedcentral.com/下载100个pdf文件，因为我正在尝试从网站下载尽可能多的生物医学pdf。我已经建立了网络刮板使用一些答案从这个网站，但我似乎不能让它正常工作

我的目标是下载PDF并将其存储在特定的文件夹中，如果您能提供帮助，我将不胜感激

url="https://jbiomedsci.biomedcentral.com/articles"
response = request.urlopen(url).read()
soup= BeautifulSoup(response, "html.parser")     
links = soup.find_all('a', href=re.compile(r'(.pdf)'))



url_list = []
  for el in links:
if(el['href'].startswith('http')):
url_list.append(el['href'])
   else:
    url_list.append("https://jbiomedsci.biomedcentral.com" + el['href'])

    print(url_list)



for url in url_list:
print(url)
pathname ="C:/Users/SciencePDF/"
fullfilename = os.path.join(pathname, url.replace("https://jbiomedsci.biomedcentral.com/articles", 
 ""))
print(fullfilename)
request.urlretrieve(url, fullfilename)

Answer 1

我已经修改了你的脚本使它工作。当您尝试以下脚本时，它将在脚本所在的同一目录中创建一个文件夹，并将下载的pdf文件存储在新创建的文件夹中

import os
import requests
from bs4 import BeautifulSoup

base = 'https://jbiomedsci.biomedcentral.com{}'
url = 'https://jbiomedsci.biomedcentral.com/articles'

res = requests.get(url)
soup = BeautifulSoup(res.text,"html.parser")

foldername = url.split("/")[-1]
os.mkdir(foldername)

for pdf in soup.select("a[data-track-action='Download PDF']"):
    filename = pdf['href'].split("/")[-1]
    fdf_link = base.format(pdf['href']) + ".pdf"
    with open(f"{foldername}/{filename}.pdf", 'wb') as f:
        f.write(requests.get(fdf_link).content)