我怎么用python在网页上刮擦？

Question

我使用Python来抓取this webpage。但是当你点击“上一个”按钮时，我就拿不下这些东西了。我想用硒来达到这个目的，但我没有成功地使它无头。在

通过下面的代码，我可以抓取匹配链接：

import urllib2
import re

site_url = 'http://us.soccerway.com'
national_league_div_sub_matches_url = 'http://us.soccerway.com/national/england/premier-league/20132014/regular-season/r21322/'
national_league_div_sub_matches_url_source = urllib2.urlopen(national_league_div_sub_matches_url).read()
match_links = re.findall('(/matches/[0-9][0-9][0-9][0-9]/.*?ICID.*?)">', national_league_div_sub_matches_url_source)
match_links = map(lambda x: ''.join([site_url, x]), match_links)
for x in match_links:
    print x

Answer 1

当您在浏览器中单击previous时，javascript调用这个长的url从服务器获取JSON数据，所以也要这样做。在

import requests, json

url = 'http://us.soccerway.com/a/block_competition_matches_summary?block_id=page_competition_1_block_competition_matches_summary_6&callback_params=%7B%22page%22%3A0%2C%22bookmaker_urls%22%3A%7B%2213%22%3A%5B%7B%22link%22%3A%22http%3A%2F%2Fwww.bet365.com%2Fhome%2F%3Faffiliate%3D365_308124%22%2C%22name%22%3A%22Bet%20365%22%7D%5D%7D%2C%22block_service_id%22%3A%22competition_summary_block_competitionmatchessummary%22%2C%22round_id%22%3A21322%2C%22outgroup%22%3Afalse%2C%22view%22%3A2%7D&action=changePage&params=%7B%22page%22%3A-1%7D'

r = requests.get(url)

#print r.text

data = json.loads(r.text)

print data

现在你有dict在data中，所以你必须找到你需要的东西。在

可能每次点击broswer中的previous都会改变url，所以如果你想得到更旧的数据，你也必须这样做。在