<p>我使用了请求,但是通过包含用户代理,您应该可以获得100多个链接。你知道吗</p>
<pre><code>import requests
from bs4 import BeautifulSoup
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'}
url = "https://www.sciencedirect.com/search?qs=hydrogen&show=25&sortBy=date&years=2018"
html = requests.get(url, headers = headers)
soup = BeautifulSoup(html.text, "html.parser")
links = soup.find_all('a')
count = 0
for link in soup.find_all('a'):
count += 1
print(link.get('href'))
print(count)
</code></pre>
<p><strong>输出:</strong></p>
<pre><code>#main_content
/
/browse/journals-and-books
/user/register?returnURL=%2Fsearch%3Fqs%3Dhydrogen%26show%3D25%26sortBy%3Ddate%26years%3D2018
/user/login?returnURL=%2Fsearch%3Fqs%3Dhydrogen%26show%3D25%26sortBy%3Ddate%26years%3D2018
https://service.elsevier.com/app/answers/detail/a_id/15904/supporthub/sciencedirect/
/browse/journals-and-books
/user/register?returnURL=%2Fsearch%3Fqs%3Dhydrogen%26show%3D25%26sortBy%3Ddate%26years%3D2018
/user/login?returnURL=%2Fsearch%3Fqs%3Dhydrogen%26show%3D25%26sortBy%3Ddate%26years%3D2018
https://service.elsevier.com/app/answers/detail/a_id/15904/supporthub/sciencedirect/
/search/advanced
/search?qs=hydrogen&show=25&sortBy=date
/search?qs=hydrogen&show=25&sortBy=date
/?qs=hydrogen&show=25&sortBy=relevance&years=2018
/search?qs=hydrogen&show=25&sortBy=relevance&years=2018
/science/article/pii/S0009250918305815
/science/journal/00092509
/science/article/pii/S0009250918305815
/science/article/pii/S0169433218321731
/science/journal/01694332
/science/article/pii/S0169433218321731
https://service.elsevier.com/app/answers/detail/a_id/27714/supporthub/sciencedirect/kw/register/
/science/article/pii/S0009250918303099
/science/journal/00092509
/science/article/pii/S0009250918303099
/science/article/pii/S0169433218322670
/science/journal/01694332
/science/article/pii/S0169433218322670
/science/article/pii/S0169433218321251
/science/journal/01694332
/science/article/pii/S0169433218321251
/science/article/pii/S1878535218302673
/science/journal/18785352
/science/article/pii/S1878535218302673/pdfft?md5=82c344dc5e6a16651e226289299ccd96&pid=1-s2.0-S1878535218302673-main.pdf
/science/article/pii/S0003267018309784
/science/journal/00032670
/science/article/pii/S0003267018309784/pdfft?md5=e84a9680b080d3521ae51a10a70e9b74&pid=1-s2.0-S0003267018309784-main.pdf
/science/article/pii/S0009250918306183
/science/journal/00092509
/science/article/pii/S0009250918306183
/science/article/pii/S0378775318311868
/science/journal/03787753
/science/article/pii/S0378775318311868
/science/article/pii/S0169433218322773
/science/journal/01694332
/science/article/pii/S0169433218322773
/science/article/pii/S0009250918305451
/science/journal/00092509
/science/article/pii/S0009250918305451
/science/article/pii/S0958694618302759
/science/journal/09586946
/science/article/pii/S0958694618302759
/science/article/pii/S0944711318306378
/science/journal/09447113
/science/article/pii/S0944711318306378
/science/article/pii/S0360319918338710
/science/journal/03603199
/science/article/pii/S0360319918338710
/science/article/pii/S109727651830981X
/science/journal/10972765
/science/article/pii/S109727651830981X
/science/article/pii/S0169433218323298
/science/journal/01694332
/science/article/pii/S0169433218323298
/science/article/pii/S0169433218322232
/science/journal/01694332
/science/article/pii/S0169433218322232
/science/article/pii/S0169433218322025
/science/journal/01694332
/science/article/pii/S0169433218322025
/science/article/pii/S0169433218335943
/science/journal/01694332
/science/article/pii/S0169433218335943
/science/article/pii/S1226086X18307378
/science/journal/1226086X
/science/article/pii/S1226086X18307378
/science/article/pii/S0169433218322372
/science/journal/01694332
/science/article/pii/S0169433218322372
/science/article/pii/S0009250918305980
/science/journal/00092509
/science/article/pii/S0009250918305980
/science/article/pii/S0169433218322955
/science/journal/01694332
/science/article/pii/S0169433218322955
/science/article/pii/S092058611831527X
/science/journal/09205861
/science/article/pii/S092058611831527X/pdfft?md5=f7c6523835be4ded224fbc28036d7218&pid=1-s2.0-S092058611831527X-main.pdf
/science/article/pii/S1878535218302661
/science/journal/18785352
/science/article/pii/S1878535218302661/pdfft?md5=2aa04be5459c3d92b5b8e7475b075146&pid=1-s2.0-S1878535218302661-main.pdf
/search?qs=hydrogen&show=50&sortBy=date&years=2018
/search?qs=hydrogen&show=100&sortBy=date&years=2018
/search?qs=hydrogen&show=25&sortBy=date&years=2018&offset=25
#
https://www.elsevier.com/
https://www.elsevier.com/solutions/sciencedirect
/customer/authenticate/manra
/science?_ob=ShoppingCartURL&_method=display&md5=3ff44acb300f01481824c54a2973d019
https://service.elsevier.com/app/contact/supporthub/sciencedirect/
https://www.elsevier.com/legal/elsevier-website-terms-and-conditions
https://www.elsevier.com/legal/privacy-policy
https://www.sciencedirect.com/legal/use-of-cookies
https://www.relx.com/
104
</code></pre>
<p>如果您仍然想使用urllib,只需做一些小的修改:</p>
<pre><code>from bs4 import BeautifulSoup
import ssl
from urllib.request import Request, urlopen
# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36'}
url = "https://www.sciencedirect.com/search?qs=hydrogen&show=25&sortBy=date&years=2018"
req = Request(url, headers=headers)
html = urlopen(req, context=ctx).read().decode('utf-8')
soup = BeautifulSoup(html, "html.parser")
count = 0
for link in soup.find_all('a'):
count += 1
print(link.get('href'))
print(count)
</code></pre>