使用JobboleSpider' object has no attribute 'browser'

来源:9-4 selenium集成到scrapy中

精慕门9737150

2017-07-07

使用'ArticleSpider.middlewares.JSPageMiddleware': 
产生问题object has no attribute 'browser'


2017-07-07 05:08:50 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)

2017-07-07 05:08:50 [scrapy.core.scraper] ERROR: Error downloading <GET http://blog.jobbole.com/all-posts/>

Traceback (most recent call last):

  File "C:\Users\sarah\Envs\artical_spider\lib\site-packages\twisted\internet\defer.py", line 1301, in _inlineCallbacks

    result = g.send(result)

  File "C:\Users\sarah\Envs\artical_spider\lib\site-packages\scrapy\core\downloader\middleware.py", line 37, in process_request

    response = yield method(request=request, spider=spider)

  File "C:/Users/sarah/Desktop/LearnPy/s0vkaq/ArticleSpider\ArticleSpider\middlewares.py", line 101, in process_request

    return HtmlResponse(url=spider.browser.current_url, body=spider.browser.page_source, encoding="utf-8", request=request)

AttributeError: 'JobboleSpider' object has no attribute 'browser'

2017-07-07 05:08:50 [scrapy.core.engine] INFO: Closing spider (finished)


前面的DEBUG是这样的(就是可以进入浏览器可以进入这个伯乐在线主页,但后续不能爬取)

http://szimg.mukewang.com/595ea9eb0001cce013660768.jpg


写回答

1回答

bobby

2017-07-10

是否没有在spider中初始化 browser对象 所以在后面访问的时候没有该对象

0
0

Scrapy打造搜索引擎 畅销4年的Python分布式爬虫课

带你彻底掌握Scrapy,用Django+Elasticsearch搭建搜索引擎

5795 学习 · 6290 问题

查看课程