完全没办法匹配拉勾网URL了
来源：7-4 Rule和LinkExtractor使用
梅小钰
2018-08-08
全是
2018-08-08 21:42:57 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://passport.lagou.com/login/login.html?msg=validation&uStatus=2&clientIp=117.159.15.221> from <GET https://www.lagou.com/gongsi/395045.html>
2018-08-08 21:42:57 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET https://passport.lagou.com/login/login.html?msg=validation&uStatus=2&clientIp=117.159.15.221> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
2018-08-08 21:42:57 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://passport.lagou.com/login/login.html?msg=validation&uStatus=2&clientIp=117.159.15.221> from <GET https://www.lagou.com/gongsi/164989.html>
2018-08-08 21:42:57 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://passport.lagou.com/login/login.html?msg=validation&uStatus=2&clientIp=117.159.15.221> from <GET https://www.lagou.com/gongsi/53.html>
2018-08-08 21:42:57 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://passport.lagou.com/login/login.html?msg=validation&uStatus=2&clientIp=117.159.15.221> from <GET https://www.lagou.com/gongsi/76066.html>
类似这样的重定向，Rule(LinkExtractor(allow=r'jobs/') 就直接找不到，各种不对，也不知道这门课该从哪里听了，完全没办法跟代码了，心态爆炸
1回答
-
bobby
2018-08-10
拉勾比较变态 使用的是通过ip限制你的抓取速度， 目前你需要限制一下你的抓取速度， 重启一下家里的路由器 换一下你的ip，
062018-11-23
相似问题