我想使用Scrapy登录以下网站:https://support.fortinet.com/Download/FirmwareImages.aspx
我登录有困难,因为它重定向到什么样的SAML链接。我在用FORMCREQUEST.from\响应()如文件中所述,但它不断地重定向那些无关痛痒的电话,我一无所获。你知道吗
下面是我的一些示例代码:
import scrapy
from scrapy.http import Request
class DownloadFirmwareSpider(scrapy.Spider):
name = "download_firmware"
start_urls = ['https://support.fortinet.com/Download/FirmwareImages.aspx']
def parse(self, response):
yield scrapy.FormRequest.from_response(
response,
formid='login_form',
formdata={
'username': 'sample_username',
'password': 'sample_password',
},
callback=self.after_login
)
def after_login(self, response):
#Specify pages to crawl here:
pagelist = ['https://support.fortinet.com/Download/FirmwareImages.aspx']
for page in pagelist:
yield Request(url= page,
callback=self.action)
def action(self, response):
# Continue processing each page...
return
以下是我尝试的日志:
2018-08-20 13:39:49 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot: fortigate)
2018-08-20 13:39:49 [scrapy.utils.log] INFO: Versions: lxml 4.2.4.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.5.0, w3lib 1.19.0, Twisted 18.7.0, Python 3.6.5 |A
naconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)], pyOpenSSL 18.0.0 (OpenSSL 1.0.2p 14 Aug 2018), cryptography 2.3.1, Platform Window
s-7-6.1.7601-SP1
2018-08-20 13:39:49 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'fortigate', 'NEWSPIDER_MODULE': 'fortigate.spiders', 'ROBOTSTXT_OBEY': True, 'SPID
ER_MODULES': ['fortigate.spiders']}
2018-08-20 13:39:49 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.logstats.LogStats']
2018-08-20 13:39:49 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2018-08-20 13:39:49 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2018-08-20 13:39:49 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2018-08-20 13:39:49 [scrapy.core.engine] INFO: Spider opened
2018-08-20 13:39:49 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-08-20 13:39:49 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-08-20 13:39:50 [scrapy.core.engine] DEBUG: Crawled (404) <GET https://support.fortinet.com/robots.txt> (referer: None)
2018-08-20 13:39:50 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://support.fortinet.com/SSO/login.ashx?ReturnUrl=%2fDownload%2
fFirmwareImages.aspx> from <GET https://support.fortinet.com/Download/FirmwareImages.aspx>
2018-08-20 13:39:50 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://support.fortinet.com/SSO/login.ashx?ReturnUrl=%2fDownload%2fFirmwareImages.aspx> (ref
erer: None)
2018-08-20 13:39:50 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (301) to <GET https://support.fortinet.com> from <GET https://customersso1.fortin
et.com/robots.txt>
2018-08-20 13:39:51 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://support.fortinet.com> (referer: None)
2018-08-20 13:39:51 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://customersso1.fortinet.com/login/saml-idp-auth/?prefix=jrx0g
5n1etn0aoy9&next=%2Fsaml-idp%2Fjrx0g5n1etn0aoy9%2Flogin_proces%2F> from <POST https://customersso1.fortinet.com/saml-idp/jrx0g5n1etn0aoy9/login/>
2018-08-20 13:39:51 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://customersso1.fortinet.com/login/saml-idp-auth/?prefix=jrx0g5n1etn0aoy9&next=%2Fsaml-i
dp%2Fjrx0g5n1etn0aoy9%2Flogin_proces%2F> (referer: https://support.fortinet.com/SSO/login.ashx?ReturnUrl=%2fDownload%2fFirmwareImages.aspx)
2018-08-20 13:39:51 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://support.fortinet.com/SSO/login.ashx?ReturnUrl=%2fDownload%2
fFirmwareImages.aspx> from <GET https://support.fortinet.com/Download/FirmwareImages.aspx>
2018-08-20 13:39:52 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://support.fortinet.com/SSO/login.ashx?ReturnUrl=%2fDownload%2fFirmwareImages.aspx> (ref
erer: https://customersso1.fortinet.com/login/saml-idp-auth/?prefix=jrx0g5n1etn0aoy9&next=%2Fsaml-idp%2Fjrx0g5n1etn0aoy9%2Flogin_proces%2F)
2018-08-20 13:39:52 [scrapy.core.engine] INFO: Closing spider (finished)
2018-08-20 13:39:52 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 4961,
'downloader/request_count': 9,
'downloader/request_method_count/GET': 8,
'downloader/request_method_count/POST': 1,
'downloader/response_bytes': 138281,
'downloader/response_count': 9,
'downloader/response_status_count/200': 4,
'downloader/response_status_count/301': 1,
'downloader/response_status_count/302': 3,
'downloader/response_status_count/404': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2018, 8, 20, 17, 39, 52, 169836),
'log_count/DEBUG': 10,
'log_count/INFO': 7,
'request_depth_max': 2,
'response_received_count': 5,
'scheduler/dequeued': 6,
'scheduler/dequeued/memory': 6,
'scheduler/enqueued': 6,
'scheduler/enqueued/memory': 6,
'start_time': datetime.datetime(2018, 8, 20, 17, 39, 49, 785700)}
2018-08-20 13:39:52 [scrapy.core.engine] INFO: Spider closed (finished)
问题是重定向页面需要Javascript,所以我必须单击“继续”按钮。我使用多个FormRequests来模拟单击按钮,直到进入正确的登录页面(进行了两次模拟单击)。你知道吗
相关问题 更多 >
编程相关推荐