如何加速Python请求

Question

因此，我有一段代码可以删除javascript内容：

from requests_html import HTMLSession

#create the session
session = HTMLSession()

#define our URL
url = 'https://partalert.net/product.js?asin=B08L8LG4M3&price=%E2%82%AC702.07&smid=A3JWKAKR8XB7XF&tag=partalertde-21&timestamp=16%3A33+UTC+%2821.4.2021%29&title=ASUS+DUAL+NVIDIA+GeForce+RTX+3070+OC+Edition+Gaming+Grafikkarte+%28PCIe+4.0%2C+8+GB+GDDR6+Speicher%2C+HDMI+2.1%2C+DisplayPort+1.4a%2C+Axial-tech+L%C3%BCfterdesign%2C+Dual+BIOS%2C+Schutzr%C3%BCckwand%2C+GPU+Tweak+II%29&tld=.de'

#use the session to get the data
r = session.get(url)

#Render the page, up the number on scrolldown to page down multiple times on a page
r.html.render(sleep=0, keep_page=True, scrolldown=0)

#take the rendered html and find the element that we are interested in
links = r.html.find('#href')

#loop through those elements extracting the text and link
for item in links:
    link = {
        'link': item.absolute_links
    }
print(link)

然而，它需要2-3秒，这是很长的方式来加载我。有没有办法加快速度

Answer 1

根本不需要刮去场地。查看源代码时，您可以看到javascript正在从输入url生成Amazon url：

document.getElementById(
          "href"
        ).href = `https://www.amazon${tld}/dp/${asin}?tag=${tag}&linkCode=ogi&th=1&psc=1&smid=${smid}`;

这意味着您只需在python中复制此函数即可生成URL。您可以使用urllib.parse获取url参数的值，然后使用字符串格式生成新url：

from urllib.parse import urlsplit, parse_qs

url = 'https://partalert.net/product.js?asin=B08L8LG4M3&price=%E2%82%AC702.07&smid=A3JWKAKR8XB7XF&tag=partalertde-21&timestamp=16%3A33+UTC+%2821.4.2021%29&title=ASUS+DUAL+NVIDIA+GeForce+RTX+3070+OC+Edition+Gaming+Grafikkarte+%28PCIe+4.0%2C+8+GB+GDDR6+Speicher%2C+HDMI+2.1%2C+DisplayPort+1.4a%2C+Axial-tech+L%C3%BCfterdesign%2C+Dual+BIOS%2C+Schutzr%C3%BCckwand%2C+GPU+Tweak+II%29&tld=.de'
query = urlsplit(url).query
params = parse_qs(query)
amazon_url = f"https://www.amazon{params['tld'][0]}/dp/{params['asin'][0]}?tag={params['tag'][0]}&linkCode=ogi&th=1&psc=1&smid={params['smid'][0]}"

结果:

https://www.amazon.de/dp/B08L8LG4M3?tag=partalertde-21&linkCode=ogi&th=1&psc=1&smid=A3JWKAKR8XB7XF