尽管URL是sam，但仍会抛出“达到最大重定向”

Question

出于一些不寻常的原因，Scrapy总是抛出“达到的最大重定向数”，即使它所说的重定向到的URL是相同的。。。在

In [1]: fetch('http://www.website.com/ap/001/new')
2018-03-16 20:50:06 [scrapy.core.engine] INFO: Spider opened
2018-03-16 20:50:06 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (meta refresh) to <GET http://www.website.com/ap/001/new> from <GET http://www.website.com/ap/001/new>
2018-03-16 20:50:07 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (meta refresh) to <GET http://www.website.com/ap/001/new> from <GET http://www.website.com/ap/001/new>
2018-03-16 20:50:07 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (meta refresh) to <GET http://www.website.com/ap/001/new> from <GET http://www.website.com/ap/001/new>
2018-03-16 20:50:07 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (meta refresh) to <GET http://www.website.com/ap/001/new> from <GET http://www.website.com/ap/001/new>
2018-03-16 20:50:07 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (meta refresh) to <GET http://www.website.com/ap/001/new> from <GET http://www.website.com/ap/001/new>
2018-03-16 20:50:07 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (meta refresh) to <GET http://www.website.com/ap/001/new> from <GET http://www.website.com/ap/001/new>
2018-03-16 20:50:07 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (meta refresh) to <GET http://www.website.com/ap/001/new> from <GET http://www.website.com/ap/001/new>
2018-03-16 20:50:07 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (meta refresh) to <GET http://www.website.com/ap/001/new> from <GET http://www.website.com/ap/001/new>
2018-03-16 20:50:07 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (meta refresh) to <GET http://www.website.com/ap/001/new> from <GET http://www.website.com/ap/001/new>
2018-03-16 20:50:07 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (meta refresh) to <GET http://www.website.com/ap/001/new> from <GET http://www.website.com/ap/001/new>
2018-03-16 20:50:07 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (meta refresh) to <GET http://www.website.com/ap/001/new> from <GET http://www.website.com/ap/001/new>
2018-03-16 20:50:07 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (meta refresh) to <GET http://www.website.com/ap/001/new> from <GET http://www.website.com/ap/001/new>
2018-03-16 20:50:07 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (meta refresh) to <GET http://www.website.com/ap/001/new> from <GET http://www.website.com/ap/001/new>
2018-03-16 20:50:07 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (meta refresh) to <GET http://www.website.com/ap/001/new> from <GET http://www.website.com/ap/001/new>
2018-03-16 20:50:07 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (meta refresh) to <GET http://www.website.com/ap/001/new> from <GET http://www.website.com/ap/001/new>
2018-03-16 20:50:08 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (meta refresh) to <GET http://www.website.com/ap/001/new> from <GET http://www.website.com/ap/001/new>
2018-03-16 20:50:08 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (meta refresh) to <GET http://www.website.com/ap/001/new> from <GET http://www.website.com/ap/001/new>
2018-03-16 20:50:08 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (meta refresh) to <GET http://www.website.com/ap/001/new> from <GET http://www.website.com/ap/001/new>
2018-03-16 20:50:08 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (meta refresh) to <GET http://www.website.com/ap/001/new> from <GET http://www.website.com/ap/001/new>
2018-03-16 20:50:08 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (meta refresh) to <GET http://www.website.com/ap/001/new> from <GET http://www.website.com/ap/001/new>
2018-03-16 20:50:08 [scrapy.downloadermiddlewares.redirect] DEBUG: Discarding <GET http://www.website.com/ap/001/new>: max redirections reached

如果我用请求和urllib2进行刮取，但是scrapy失败了。。。是什么导致了这个问题，我该如何解决它。我在stackoverflow/google上找不到任何与此问题匹配的内容。在

我已经检查了我的浏览器和“重定向检查程序”的网址，他们都不认为该网站是重定向，所以不知道为什么scrapy认为它是重定向。在