我正在尝试使用scrapy splash从这个website获取数据,但我无法提取数据。我想获取每个真实状态的数据,如href、price等。以下是我的代码:
在settings.py中:
ROBOTSTXT_OBEY = False
USER_AGENT = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36"
SPLASH_ENABLED = True
DOWNLOADER_MIDDLEWARES = {
'scrapy_splash.SplashCookiesMiddleware': 723,
'scrapy_splash.SplashMiddleware': 725,
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810,
}
SPLASH_URL = 'http://localhost:8050/'
SPIDER_MIDDLEWARES = {
'scrapy_splash.SplashDeduplicateArgsMiddleware': 100,
}
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
HTTPCACHE_STORAGE = 'scrapy_splash.SplashAwareFSCacheStorage'
我的蜘蛛:
class M2Spider(scrapy.Spider):
name = "m2"
allowed_domains = ['metrocuadrado.com']
start_urls = [
'https://www.metrocuadrado.com/bodega/arriendo'
]
def start_requests(self):
for url in self.start_urls:
yield SplashRequest(url=url,callback= self.parse,
endpoint='render.html',
args={'wait': 10},)
def parse(self, response):
print("--------------------------------------------------------------")
real_states= response.selector.xpath(".//a[@class='sc-bdVaJa ebNrSm']").getall()
print("real_states")
输出打印为空列表[]。我是个新手。有什么建议吗
我要做的是:
向https://www.metrocuadrado.com/results/_next/static/chunks/commons.8afec6af6d5add2097bf.js发送请求,如果搜索“X-API-key”,您将在响应中找到一个API键。这样就可以很容易地用正则表达式提取,比如:
re.findall(r'"X-Api-Key":"(\w+)"')
然后,提取API密钥后,向https://www.metrocuadrado.com/rest-search/search?seo=/bodega/arriendo&from=0&size=50发送请求,这是您发送的网站中隐藏的API。要获得有效的响应,必须像这样附加标题
通过该API,您可以获得JSON格式的数据,这通常比解析html更可靠,因为这会更频繁地发生变化
相关问题 更多 >
编程相关推荐