在抓取第一页之前，如何在不知道曲奇是什么的情况下传递饼干？

Question

我想从多个页面获取数据。如果我想从第二页获取数据，我应该使用cookies传递搜索项（因为搜索项不会出现在url中）
第一页的url是：

http://epgd.biosino.org/EPGD/search/textsearch.jsp?textquery=man&submit=Feeling+Lucky

第二页的url是：

http://epgd.biosino.org/EPGD/search/textsearch.jsp?currentIndex=10

我在堆栈溢出中看到了很多问题，他们在爬网之前都知道cookies是什么数据。但是只有当我完成第一页的爬行后，我才能得到饼干。所以我想知道怎么处理这个？ 这是我的代码：

__author__ = 'Rabbit'
from scrapy.spiders import Spider
from scrapy.selector import Selector

from scrapy_Data.items import EPGD


class EPGD_spider(Spider):
    name = "EPGD"
    allowed_domains = ["epgd.biosino.org"]
    stmp = []
    term = "man"
    url_base = "http://epgd.biosino.org/EPGD/search/textsearch.jsp?textquery=man&submit=Feeling+Lucky"

    start_urls = stmp

    def parse(self, response):
        sel = Selector(response)
        sites = sel.xpath('//tr[@class="odd"]|//tr[@class="even"]')

        for site in sites:
            item = EPGD()
            item['genID'] = map(unicode.strip, site.xpath('td[1]/a/text()').extract())
            item['taxID'] = map(unicode.strip, site.xpath('td[2]/a/text()').extract())
            item['familyID'] = map(unicode.strip, site.xpath('td[3]/a/text()').extract())
            item['chromosome'] = map(unicode.strip, site.xpath('td[4]/text()').extract())
            item['symbol'] = map(unicode.strip, site.xpath('td[5]/text()').extract())
            item['description'] = map(unicode.strip, site.xpath('td[6]/text()').extract())
            yield item

Answer 1

Scrapy接收并跟踪服务器发送的cookie，并在后续请求时发送它们，就像任何普通的web浏览器一样，检查更多信息here

我看不出您是如何在代码上分页的，但应该是这样的：

class EPGD_spider(Spider):
    name = "EPGD"
    allowed_domains = ["epgd.biosino.org"]
    stmp = []
    term = "man"
    my_urls = ["http://epgd.biosino.org/EPGD/search/textsearch.jsp?textquery=man&submit=Feeling+Lucky"]

    def parse(self, response):
        sel = Selector(response)
        sites = sel.xpath('//tr[@class="odd"]|//tr[@class="even"]')

        for site in sites:
            item = EPGD()
            item['genID'] = map(unicode.strip, site.xpath('td[1]/a/text()').extract())
            item['taxID'] = map(unicode.strip, site.xpath('td[2]/a/text()').extract())
            item['familyID'] = map(unicode.strip, site.xpath('td[3]/a/text()').extract())
            item['chromosome'] = map(unicode.strip, site.xpath('td[4]/text()').extract())
            item['symbol'] = map(unicode.strip, site.xpath('td[5]/text()').extract())
            item['description'] = map(unicode.strip, site.xpath('td[6]/text()').extract())
            yield item
        yield Request('http://epgd.biosino.org/EPGD/search/textsearch.jsp?currentIndex=10', 
                       callback=self.parse_second_url)

     def parse_second_url(self, response):
         # do your thing

第二个请求携带第一个请求的cookies。在