downloadmiddleware随机更换user-agent不能调用process_request
来源:8-5 通过downloadmiddleware随机更换user-agent - 2
qq_雨过天晴_2
2017-07-31
老师您好,在用downloadmiddleware随机更换user-agent时我定义了函数RandomUserAgentMiddlware,并在settings文件中配置了,但是调试时程序只执行了__init__函数,没有调用process_request函数,不知道是什么原因
class RandomUserAgentMiddlware(object):
#随机更换user-agent
def __init__(self, crawler):
super(RandomUserAgentMiddlware, self).__init__()
self.ua = UserAgent()
self.ua_type = crawler.settings.get("RANDOM_UA_TYPE", "random")
@classmethod
def from_crawler(cls, crawler):
return cls(crawler)
def process_request(self, request, spider):
def get_ua():
return getattr(self.ua, self.ua_type)
request.headers.setdefault('User-Agent', get_ua())
SPIDER_MIDDLEWARES = {
'MyArticleSpider.middlewares.RandomUserAgentMiddlware': 1,
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None, #需要取消
}写回答
1回答
-
qq_雨过天晴_2
提问者
2017-07-31
找到错误了,SPIDER_MIDDLEWARES配置错了,应该配置DOWNLOADER_MIDDLEWARES
012017-08-01
相似问题