无法debug 到 parse_detail 该怎么解决

来源：1-1 python分布式爬虫打造搜索引擎简介

qq_漫漫水天_0

2022-03-27

from urllib import parse
from scrapy.http import Request

import scrapy
import undetected_chromedriver.v2 as uc
class JobboleSpider(scrapy.Spider):
name = 'jobbole’
allowed_domains = [‘news.cnblogs.com’]
start_urls = [‘http://news.cnblogs.com/’]
custom_settings = {
#使后面请求沿用之前的cookie,默认是关闭的，这个设置只对这个爬虫生效
"COOKIES_ENABLED":True
}
def start_requests(self):
#这是入口可以模拟登陆拿到cookie,selenium 控制浏览器会被有些网站识别出来，undetected-chromedriver可以防止被识别
browser = uc.Chrome()
browser.get(“https://account.cnblogs.com/signin”)
input(“回车继续：”)
cookies = browser.get_cookies()
cookie_dic ={}
for cookie in cookies:
cookie_dic[cookie[‘name’]]= cookie[“value”]
for url in self.start_urls:
#将cookie交给scrapy,
headers = {
“user-agent”:“Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36”
}
yield scrapy.Request(url, headers=headers, cookies=cookie_dic, callback=self.parse,dont_filter=True)

def parse(self, response):
    #获取新闻列表url并传递给scrap下载后调用相应的处理方法
    #获取下一页的url并传递scrapy下载，并交给此parese处理


    # #获取新闻列表url和对应的图片
    post_nodes= response.xpath("//div[@id='news_list']/div[@class='news_block']")
    for post_node in post_nodes:
        image_url = post_node.xpath("//div[@class='entry_summary']/a/img/@src").extract_first("")
        post_url = post_node.xpath("//h2[@class='news_entry']/a/@href").extract_first("")
        # 交出url
        yield Request(url=parse.urljoin(response.url, post_url), meta={"front_image_url": image_url},
                      callback=self.parse_detail,dont_filter=True)

#     pass
# #详情页解析
def parse_detail(self, response):
    pass

C:\Users\acer\scrapytest\article_spider\Scripts\python.exe “D:\Program Files\JetBrains\PyCharm Community Edition 2021.2\plugins\python-ce\helpers\pydev\pydevd.py” --multiproc --qt-support=auto --client 127.0.0.1 --port 57674 --file C:/Users/acer/scrapytest/article_spider/Scripts/ArticleSpider/main.py
Connected to pydev debugger (build 212.4746.96)
2022-03-27 15:09:32 [scrapy.utils.log] INFO: Scrapy 2.6.1 started (bot: ArticleSpider)
2022-03-27 15:09:32 [scrapy.utils.log] INFO: Versions: lxml 4.8.0.0, libxml2 2.9.12, cssselect 1.1.0, parsel 1.6.0, w3lib 1.22.0, Twisted 22.2.0, Python 3.7.6 (tags/v3.7.6:43364a7ae0, Dec 19 2019, 00:42:30) [MSC v.1916 64 bit (AMD64)], pyOpenSSL 22.0.0 (OpenSSL 1.1.1n 15 Mar 2022), cryptography 36.0.2, Platform Windows-7-6.1.7601-SP1
2022-03-27 15:09:32 [scrapy.crawler] INFO: Overridden settings:
{‘BOT_NAME’: ‘ArticleSpider’,
‘NEWSPIDER_MODULE’: ‘ArticleSpider.spiders’,
‘ROBOTSTXT_OBEY’: True,
‘SPIDER_MODULES’: [‘ArticleSpider.spiders’]}
2022-03-27 15:09:32 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.selectreactor.SelectReactor
2022-03-27 15:09:32 [scrapy.extensions.telnet] INFO: Telnet Password: bf563b396e3bbe26
2022-03-27 15:09:32 [scrapy.middleware] INFO: Enabled extensions:
[‘scrapy.extensions.corestats.CoreStats’,
‘scrapy.extensions.telnet.TelnetConsole’,
‘scrapy.extensions.logstats.LogStats’]
2022-03-27 15:09:33 [scrapy.middleware] INFO: Enabled downloader middlewares:
[‘scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware’,
‘scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware’,
‘scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware’,
‘scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware’,
‘scrapy.downloadermiddlewares.useragent.UserAgentMiddleware’,
‘scrapy.downloadermiddlewares.retry.RetryMiddleware’,
‘scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware’,
‘scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware’,
‘scrapy.downloadermiddlewares.redirect.RedirectMiddleware’,
‘scrapy.downloadermiddlewares.cookies.CookiesMiddleware’,
‘scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware’,
‘scrapy.downloadermiddlewares.stats.DownloaderStats’]
2022-03-27 15:09:33 [scrapy.middleware] INFO: Enabled spider middlewares:
[‘scrapy.spidermiddlewares.httperror.HttpErrorMiddleware’,
‘scrapy.spidermiddlewares.offsite.OffsiteMiddleware’,
‘scrapy.spidermiddlewares.referer.RefererMiddleware’,
‘scrapy.spidermiddlewares.urllength.UrlLengthMiddleware’,
‘scrapy.spidermiddlewares.depth.DepthMiddleware’]
2022-03-27 15:09:33 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2022-03-27 15:09:33 [scrapy.core.engine] INFO: Spider opened
2022-03-27 15:09:33 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-03-27 15:09:33 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2022-03-27 15:09:33 [undetected_chromedriver.patcher] DEBUG: getting release number from /LATEST_RELEASE
2022-03-27 15:09:34 [undetected_chromedriver.patcher] DEBUG: downloading from https://chromedriver.storage.googleapis.com/99.0.4844.51/chromedriver_win32.zip
2022-03-27 15:09:37 [undetected_chromedriver.patcher] DEBUG: unzipping C:\Users\acer\AppData\Local\Temp\tmpyu_ia90k
2022-03-27 15:09:37 [undetected_chromedriver.patcher] INFO: patching driver executable C:\Users\acer\appdata\roaming\undetected_chromedriver\80b8ecd3a2c73f72_chromedriver.exe
2022-03-27 15:09:43 [selenium.webdriver.remote.remote_connection] DEBUG: POST http://localhost:57707/session {“capabilities”: {“firstMatch”: [{}], “alwaysMatch”: {“browserName”: “chrome”, “pageLoadStrategy”: “normal”, “goog:chromeOptions”: {“extensions”: [], “binary”: “C:\Users\acer\AppData\Local\Google\Chrome\Application\chrome.exe”, “args”: ["–remote-debugging-host=127.0.0.1", “–remote-debugging-port=57696”, “–user-data-dir=C:\Users\acer\AppData\Local\Temp\tmp01r030n1”, “–lang=zh-CN”, “–no-default-browser-check”, “–no-first-run”, “–log-level=0”], “debuggerAddress”: “127.0.0.1:57696”}}}, “desiredCapabilities”: {“browserName”: “chrome”, “pageLoadStrategy”: “normal”, “goog:chromeOptions”: {“extensions”: [], “binary”: “C:\Users\acer\AppData\Local\Google\Chrome\Application\chrome.exe”, “args”: ["–remote-debugging-host=127.0.0.1", “–remote-debugging-port=57696”, “–user-data-dir=C:\Users\acer\AppData\Local\Temp\tmp01r030n1”, “–lang=zh-CN”, “–no-default-browser-check”, “–no-first-run”, “–log-level=0”], “debuggerAddress”: “127.0.0.1:57696”}}}
2022-03-27 15:09:43 [urllib3.connectionpool] DEBUG: Starting new HTTP connection (1): localhost:57707
2022-03-27 15:09:44 [urllib3.connectionpool] DEBUG: http://localhost:57707 “POST /session HTTP/1.1” 200 666
2022-03-27 15:09:44 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
2022-03-27 15:09:44 [selenium.webdriver.remote.remote_connection] DEBUG: POST http://localhost:57707/session/8917584b0defd9f38e158a95d7ebf35d/execute/sync {“script”: "\n let objectToInspect = window,\n result = [];\n while(objectToInspect !== null)\n { result = result.concat(Object.getOwnPropertyNames(objectToInspect));\n objectToInspect = Object.getPrototypeOf(objectToInspect); }\n return result.filter(i => i.match(/.+.+(Array|Promise|Symbol)/ig))\n ", “args”: []}
2022-03-27 15:09:44 [urllib3.connectionpool] DEBUG: http://localhost:57707 “POST /session/8917584b0defd9f38e158a95d7ebf35d/execute/sync HTTP/1.1” 200 119
2022-03-27 15:09:44 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
2022-03-27 15:09:44 [selenium.webdriver.remote.remote_connection] DEBUG: POST http://localhost:57707/session/8917584b0defd9f38e158a95d7ebf35d/goog/cdp/execute {“cmd”: “Page.addScriptToEvaluateOnNewDocument”, “params”: {“source”: "\n let objectToInspect = window,\n result = [];\n while(objectToInspect !== null) \n { result = result.concat(Object.getOwnPropertyNames(objectToInspect));\n objectToInspect = Object.getPrototypeOf(objectToInspect); }\n result.forEach(p => p.match(/.+.+(Array|Promise|Symbol)/ig)\n &&delete window[p]&&console.log(‘removed’,p))\n "}}
2022-03-27 15:09:44 [urllib3.connectionpool] DEBUG: http://localhost:57707 “POST /session/8917584b0defd9f38e158a95d7ebf35d/goog/cdp/execute HTTP/1.1” 200 28
2022-03-27 15:09:44 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
2022-03-27 15:09:44 [selenium.webdriver.remote.remote_connection] DEBUG: POST http://localhost:57707/session/8917584b0defd9f38e158a95d7ebf35d/url {“url”: “https://account.cnblogs.com/signin”}
2022-03-27 15:09:48 [urllib3.connectionpool] DEBUG: http://localhost:57707 “POST /session/8917584b0defd9f38e158a95d7ebf35d/url HTTP/1.1” 200 14
2022-03-27 15:09:48 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
回车继续：2022-03-27 15:11:12 [selenium.webdriver.remote.remote_connection] DEBUG: GET http://localhost:57707/session/8917584b0defd9f38e158a95d7ebf35d/cookie {}
2022-03-27 15:11:12 [urllib3.connectionpool] DEBUG: http://localhost:57707 “GET /session/8917584b0defd9f38e158a95d7ebf35d/cookie HTTP/1.1” 200 1456
2022-03-27 15:11:12 [selenium.webdriver.remote.remote_connection] DEBUG: Finished Request
2022-03-27 15:11:12 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2022-03-27 15:11:12 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET https://news.cnblogs.com/robots.txt> from <GET http://news.cnblogs.com/robots.txt>
2022-03-27 15:11:12 [py.warnings] WARNING: C:\Users\acer\scrapytest\article_spider\lib\site-packages\scrapy\core\engine.py:276: ScrapyDeprecationWarning: Passing a ‘spider’ argument to ExecutionEngine.download is deprecated
return self.download(result, spider) if isinstance(result, Request) else result