<p>如果(url在两个页面之间没有变化),那么您应该使用scrapy.Request()添加<em>don_filter=True</em>,否则scrapy在处理完第一个页面后会发现此url是重复的。</p>
<p>如果需要使用javascript呈现页面,则应该使用<a href="https://blog.scrapinghub.com/2015/03/02/handling-javascript-in-scrapy-with-splash" rel="nofollow noreferrer">scrapy-splash</a>,还可以检查这个<a href="https://github.com/clemfromspace/scrapy-selenium" rel="nofollow noreferrer">scrapy middleware</a>,它可以使用selenium处理javascript页面,也可以通过启动任何无头浏览器来完成</p>
<p>但更有效、更快的解决方案是检查浏览器,查看在提交表单或触发某个事件期间发出的请求。尝试模拟浏览器发送的相同请求。如果您能够正确地复制请求,您将获得所需的数据。</p>
<p>下面是一个例子:</p>
<pre><code>class ScrollScraper(Spider):
name = "scrollingscraper"
quote_url = "http://quotes.toscrape.com/api/quotes?page="
start_urls = [quote_url + "1"]
def parse(self, response):
quote_item = QuoteItem()
print response.body
data = json.loads(response.body)
for item in data.get('quotes', []):
quote_item["author"] = item.get('author', {}).get('name')
quote_item['quote'] = item.get('text')
quote_item['tags'] = item.get('tags')
yield quote_item
if data['has_next']:
next_page = data['page'] + 1
yield Request(self.quote_url + str(next_page))
</code></pre>
<p>当每一页的分页url相同时&;使用POST-request,则可以使用<em>scrapy.FormRequest()</em>而不是<em>scrapy.request()</em>,两者相同,但FormRequest向构造函数添加了一个新参数(<em>formdata=</em>)。</p>
<p>下面是这个<a href="https://stackoverflow.com/questions/30189862/scrapy-follow-pagination-ajax-request-post">post</a>中的另一个蜘蛛示例:</p>
<pre><code>class SpiderClass(scrapy.Spider):
# spider name and all
name = 'ajax'
page_incr = 1
start_urls = ['http://www.pcguia.pt/category/reviews/#paginated=1']
pagination_url = 'http://www.pcguia.pt/wp-content/themes/flavor/functions/ajax.php'
def parse(self, response):
sel = Selector(response)
if self.page_incr > 1:
json_data = json.loads(response.body)
sel = Selector(text=json_data.get('content', ''))
# your code here
# pagination code starts here
if sel.xpath('//div[@class="panel-wrapper"]'):
self.page_incr += 1
formdata = {
'sorter': 'recent',
'location': 'main loop',
'loop': 'main loop',
'action': 'sort',
'view': 'grid',
'columns': '3',
'paginated': str(self.page_incr),
'currentquery[category_name]': 'reviews'
}
yield FormRequest(url=self.pagination_url, formdata=formdata, callback=self.parse)
else:
return
</code></pre>