无法从DuckDuckGo搜索结果中刮取链接

Question

我想从DuckDuckGo搜索结果中删除第一个链接。我编写了以下代码：

import requests
from bs4 import BeautifulSoup
class Bse:
      def currentPrice(self,symbol):
            headers = {
                  "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:84.0) Gecko/20100101 Firefox/84.0"
            }
            duckDuckUrl=f'https://duckduckgo.com/?q=bse+{symbol}+stock+price'
            response=requests.get(duckDuckUrl,headers=headers)
            soup=BeautifulSoup(response.text,"html.parser")
             bseIndiaLink=soup.find_all('a')
            # bseIndiaLink=soup.find_all('a',class_="result__a")  #giving empty list
            print(bseIndiaLink)


bse=Bse()
bse.currentPrice('reliance')

首先，我在beautifulSoup中使用了find_all（），没有class_参数。它返回了一些随机锚标签的列表，这些标签对我没有任何用处。 我还尝试使用class_u参数查找_all（），但它返回了一个空列表

我试图打印汤对象。它打印的是网页的HTML，而不是那些包含div的结果。我不知道为什么BeautifulSoup不删除包含div的结果。请看屏幕截图，突出显示的HTML语法是我想要的：

我发现DuckDuckGo在搜索结果中使用javascript，而beautifulSoup无法获取javascript，但在StackOverflow的其他帖子中，我发现人们可以从搜索结果中获取链接。
但是如果我使用谷歌而不是DuckDuckGo，我就能够获得所需的链接

我想知道为什么我不能从DuckDuckGo上抓取，而是使用相同的代码从Google上抓取。我很好奇

如果有人知道我遗漏了什么，请告诉我。这将有助于我的学习之旅

多谢各位

Answer 1

Firstly I used find_all() in beautifulSoup without class_ argument. It returned me a list of some random anchor tags that are not of any use to me.

这是正确的行为，因为您要求bs4获取所有<a>标记，并且它返回了找到的所有<a>标记

您可以更改URL以获取非JavaScript版本：

from this (JS): https://duckduckgo.com/?q=bse+reliance+stock+price&t=hx&va=g&ia=web
to this (non-JS): https://html.duckduckgo.com/html/?q=bse%20reliance%20stock%20price

如果每次只需要先提取链接，则可以执行以下操作：

>>> first_url = soup.select_one('.result__url')['href'].replace('//', '')
"duckduckgo.com/l/?uddg=https%3A%2F%2Fwww.bseindia.com%2Fstock%2Dshare%2Dprice%2Freliance%2Dindustries%2Dltd%2Freliance%2F500325%2F&rut=b13b3c373de61ffd03dee7ad51f9fb9274dac16d098f25920d7946dbd9a73cc7"

代码和full example in the online IDE：

import requests, lxml
from bs4 import BeautifulSoup

headers = {
    'User-agent':
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}

params = {
  "q": "bse reliance stock price",
  "kl": "us-en" # language
}

html = requests.get('https://html.duckduckgo.com/html', headers=headers, params=params)
soup = BeautifulSoup(html.text, 'lxml')

first_url = soup.select_one('.result__url')['href'].replace('//', '')
print(first_url)

# duckduckgo.com/l/?uddg=https%3A%2F%2Fwww.bseindia.com%2Fstock%2Dshare%2Dprice%2Freliance%2Dindustries%2Dltd%2Freliance%2F500325%2F&rut=b13b3c373de61ffd03dee7ad51f9fb9274dac16d098f25920d7946dbd9a73cc7

---

或者，您可以使用SerpApi中的DuckDuckGo Organic Results API。这是一个免费的付费API。查看playground

不同之处在于它刮去了DuckDuckGo的JavaScript版本，唯一需要做的就是迭代JSON字符串并提取所需内容

要集成的代码：

from serpapi import GoogleSearch
import os

params = {
  "api_key": os.getenv("API_KEY"),
  "engine": "duckduckgo",
  "q": "bse reliance stock price",
  "kl": "us-en"
}

search = GoogleSearch(params)
results = search.get_dict()

# [0] - index of the first organic result
first_link = results['organic_results'][0]['link']
print(first_link)

# https://www.bseindia.com/stock-share-price/reliance-industries-ltd/reliance/500325/

Disclaimer, I work for SerpApi.

Answer 2

这将根据您当前的搜索关键字生成结果。您需要发送post http请求以及适当的参数来访问内容。为了使您当前的尝试成功，我在有效负载中使用了一些字符串格式

import requests
from bs4 import BeautifulSoup

class Bse:
    def __init__(self):
        self.duckDuckUrl = 'https://html.duckduckgo.com/html/'
        self.payload = {'q': 'bse {} stock price','b': ''}
        self.headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:84.0) Gecko/20100101 Firefox/84.0'}

    def currentPrice(self,symbol):
        self.payload['q'] = self.payload['q'].format(symbol)
        res = requests.post(self.duckDuckUrl,data=self.payload,headers=self.headers)
        soup = BeautifulSoup(res.text,'html.parser')
        return soup.find('a',class_='result__a').get("href")

if __name__ == '__main__':
    bse = Bse()
    print(bse.currentPrice('reliance'))

使用get请求：

link = "https://html.duckduckgo.com/html/?"
params = {'q': 'nse {} stock price'}

def fetch_first_link(s,symbol):
    params['q'] = params['q'].format(symbol)
    res = s.get(link,params=params)
    soup = BeautifulSoup(res.text,"lxml")
    item = soup.select_one(".result__title > a.result__a").get("href")
    return item

if __name__ == '__main__':
    with requests.Session() as s:
        s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36'
        print(fetch_first_link(s,'reliance'))