带请求的AJAX页面

Question

我想把this booking flow的结果刮下来。你知道吗

通过查看“网络”选项卡，我发现数据是通过一个ajax GET在以下URL检索的：

https://shop.caremar.it/main_acquista_1_corse_00_ajax.asp?l=it&data=24/02/2019&portoP=3&portoA=5&form_url=ticket_s1_2

我构建了传递参数的URL，如下所示：

params = urllib.parse.urlencode({
        'data': '24/02/2019',
        'portoP': '3' ,
        'portoA': '5',
        'form_url': 'ticket_s1_2',
    })

并提出请求：

caremar_timetable_url = "https://shop.caremar.it/main_acquista_1_corse_00_ajax.asp?l=it&"
print(f"https://shop.caremar.it/main_acquista_1_corse_00_ajax.asp?l=it&{params}")
headers = {'user-agent':'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.3'}
res = requests.get(caremar_timetable_url,headers=headers, params=params)
soup = BeautifulSoup(res.text,'html.parser')
print(soup.text)

输出

https://shop.caremar.it/main_acquista_1_corse_00_ajax.asp?l=it&data=24%2F02%2F2019&portoP=7&portoA=1&form_url=ticket_s1_2
Non è stato possibile procedere con l'acquisto del biglietto online. Si prega di riprovare

该网站的回复是一条错误消息，称无法完成预订。如果我复制并粘贴我在浏览器中创建的URL，我会得到一个没有样式的HTML页面，其中包含我需要的数据。 为什么会这样？我怎样才能克服它？你知道吗

Answer 1

数据似乎随请求而来

import requests
from bs4 import BeautifulSoup as bs
url = 'https://shop.caremar.it/main_acquista_1_corse_00_ajax.asp?l=it&data=27/02/2019&portoP=1&portoA=4&form_url=ticket_s1_2'

res = requests.get(url)
soup = bs(res.content, 'lxml')
print(soup.select_one('html'))