Scrapy爬虫未返回任何结果

Question

这是我第一次尝试创造一个蜘蛛，请原谅我，如果我做得不好。 这是我试图从中提取数据的网站链接。http://www.4icu.org/in/。我想要在页面上显示的所有大学的名单。但是当我运行下面的spider时，返回的是一个空的json文件。 我的项目.py在

    import scrapy
    class CollegesItem(scrapy.Item):
    # define the fields for your item here like:
        link = scrapy.Field() 

这是蜘蛛 大学.py在

^{pr2}$

Answer 1

正如问题的注释中所述，您的代码有一些问题。在

首先，您不需要两个方法，因为在parse方法中，您调用了与在start_urls中相同的URL。在

若要从站点获取一些信息，请尝试使用以下代码：

def parse(self, response):
    for tr in response.xpath('//div[@class="section group"][5]/div[@class="col span_2_of_2"][1]/table//tr'):
        if tr.xpath(".//td[@class='i']"):
            name = tr.xpath('./td[1]/a/text()').extract()[0]
            location = tr.xpath('./td[2]//text()').extract()[0]
            print name, location

并根据你的需要调整它来填充你的物品。在

如您所见，您的浏览器在table中显示了一个额外的tbody，当您使用Scrapy进行刮取时，它不存在。这意味着您经常需要判断您在浏览器中看到的内容。在

Answer 2

这是工作代码

    import scrapy
    from scrapy.spider import Spider
    from scrapy.http import Request

    class CollegesItem(scrapy.Item):
    # define the fields for your item here like:
        name = scrapy.Field()
        location = scrapy.Field()
    class CollegesSpider(Spider):
        name = 'colleges'
        allowed_domains = ["4icu.org"]
        start_urls = ('http://www.4icu.org/in/',)

        def parse(self, response):
            for tr in response.xpath('//div[@class="section group"] [5]/div[@class="col span_2_of_2"][1]/table//tr'):
                if tr.xpath(".//td[@class='i']"):
                    item = CollegesItem()
                    item['name'] = tr.xpath('./td[1]/a/text()').extract()[0]
                    item['location'] = tr.xpath('./td[2]//text()').extract()[0]
                    yield item

运行命令后 蜘蛛

^{pr2}$

以下是结果片段：

    [[[[[[[{"name": "Indian Institute of Technology Bombay", "location": "Mumbai"},
    {"name": "Indian Institute of Technology Madras", "location": "Chennai"},
    {"name": "University of Delhi", "location": "Delhi"},
    {"name": "Indian Institute of Technology Kanpur", "location": "Kanpur"},
    {"name": "Anna University", "location": "Chennai"},
    {"name": "Indian Institute of Technology Delhi", "location": "New Delhi"},
    {"name": "Manipal University", "location": "Manipal ..."},
    {"name": "Indian Institute of Technology Kharagpur", "location": "Kharagpur"},
    {"name": "Indian Institute of Science", "location": "Bangalore"},
    {"name": "Panjab University", "location": "Chandigarh"},
    {"name": "National Institute of Technology, Tiruchirappalli", "location": "Tiruchirappalli"}, .........