仅在PDF嵌入URL中删除包含特定单词的段落

Question

我目前正在开发一些代码来从网站上抓取文本。我不想刮去整个页面，只想刮去页面中包含某些单词的部分。我已经使用.find_all（“p”）命令成功地对大多数URL执行了此操作，但是这不适用于定向到PDF的URL

我似乎找不到一种方法将PDF文件作为文本打开，然后将文本分成段落。这就是我想做的：首先1）打开一个PDF嵌入的URL作为文本，2）将文本分成多个段落。这样，我就只能抓取包含某些单词的段落

下面是我目前用来为“普通”URL删除包含某些单词的段落的代码。非常感谢为PDF嵌入URL（如变量“url2”，下面的代码）提供的任何技巧

from urllib.request import Request, urlopen
from bs4 import BeautifulSoup
import re

url1 = "https://brainybackpackers.com/best-places-for-whale-watching-in-the-world/"
url2 = "https://www.environment.gov.au/system/files/resources/7f15bfc1-ed3d-40b6-a177-c81349028ef6/files/aust-national-guidelines-whale-dolphin-watching-2017.pdf"
url = url1
req = Request(url, headers={"User-Agent": 'Mozilla/5.0'})
page = urlopen(req, timeout = 5) # Open page within 5 seconds. This line skips 'empty' websites
htmlParse = BeautifulSoup(page.read(), 'lxml') 
SearchWords = ["orca", "killer whale", "humpback"] # text must contain these words

# Check if the article text mentions the SearchWord(s). If so, continue the analysis. 
if any(word in htmlParse.text for word in SearchWords):
    textP = ""
    text = ""
    
    # Look for paragraphs ("p") that contain a SearchWord
    for word in SearchWords:
        print(word)
        for para in htmlParse.find_all("p", text = re.compile(word)): 
            textParagraph = para.get_text()
            textP = textP + textParagraph
    text= text + textP
    print(text)

Answer 1

您可以阅读PDF并在页面中搜索您要查找的内容：

# pip install pyPDF2

import io
import requests
import PyPDF2


URI = "https://www.environment.gov.au/system/files/resources/7f15bfc1-ed3d-40b6-a177-c81349028ef6/files/aust-national-guidelines-whale-dolphin-watching-2017.pdf"

r = requests.get(URI)
with io.BytesIO(r.content) as f:
  reader = PyPDF2.PdfFileReader(f)
  num_pages = reader.numPages
  
  data = []
  # place page text to data
  for page in range(num_pages):
    page_data = reader.getPage(page)
    data.append(page_data.extractText())

# look up
search_words = set(["orca", "killer whale", "humpback"])

# get pages containing your lookup
wanted_page = []
for page_contents in data:
     for word in search_words:
         if word in page_contents.lower():
             wanted_page.append(page_contents)

             
print(wanted_page)