scrapyd-deloy打包的时候出现错误

来源:16-1 scrapyd部署scrapy项目

慕函数7358036

2020-04-21

scrapyd-deloy打包的时候出现错误,返回的错误信息如下

{“node_name”: “localhost.localdomain”, “status”: “error”,
“message”: “/root/PycharmProjects/Graduation_design/venv/lib/python3.7/site-packages/scrapy/utils/project.py:94:
ScrapyDeprecationWarning: Use of environment variables prefixed with SCRAPY_ to override settings is deprecated. The following environment variables are currently defined: EGG_VERSION
ScrapyDeprecationWarning
Traceback (most recent call last):
File “/usr/local/python3/lib/python3.7/runpy.py”, line 193, in _run_module_as_main
"main”, mod_spec)
File “/usr/local/python3/lib/python3.7/runpy.py”, line 85, in _run_code
exec(code, run_globals)
File “/root/PycharmProjects/Graduation_design/venv/lib/python3.7/site-packages/scrapyd/runner.py”, line 40, in
main()\n File “/root/PycharmProjects/Graduation_design/venv/lib/python3.7/site-packages/scrapyd/runner.py”, line 37, in main
execute()\n File “/root/PycharmProjects/Graduation_design/venv/lib/python3.7/site-packages/scrapy/cmdline.py”, line 144, in execute
cmd.crawler_process = CrawlerProcess(settings)
File “/root/PycharmProjects/Graduation_design/venv/lib/python3.7/site-packages/scrapy/crawler.py”, line 265, in init
super(CrawlerProcess, self).init(settings)
File “/root/PycharmProjects/Graduation_design/venv/lib/python3.7/site-packages/scrapy/crawler.py”, line 137, in init
self.spider_loader = _get_spider_loader(settings)
File “/root/PycharmProjects/Graduation_design/venv/lib/python3.7/site-packages/scrapy/crawler.py”, line 345, in _get_spider_loader
return loader_cls.from_settings(settings.frozencopy())
File “/root/PycharmProjects/Graduation_design/venv/lib/python3.7/site-packages/scrapy/spiderloader.py”, line 60, in from_settings
return cls(settings)\n File “/root/PycharmProjects/Graduation_design/venv/lib/python3.7/site-packages/scrapy/spiderloader.py”, line 24, in init
self._load_all_spiders()\n File “/root/PycharmProjects/Graduation_design/venv/lib/python3.7/site-packages/scrapy/spiderloader.py”, line 46, in _load_all_spiders
for module in walk_modules(name):
File “/root/PycharmProjects/Graduation_design/venv/lib/python3.7/site-packages/scrapy/utils/misc.py”, line 77, in walk_modules
submod = import_module(fullpath)
File “/usr/local/python3/lib/python3.7/importlib/init.py”, line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “

但是我根本不知道这缺的是个什么模块,而且在settings中我也导入环境变量了,运行scrapy list也没有问题

目录结构如下

图片描述

settings配置如下

import sys
import os

BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(file)))
sys.path.insert(0, os.path.join(BASE_DIR, ‘master_spider’))

BOT_NAME = 'master_spider’
LOG_LEVEL = “INFO”

SPIDER_MODULES = [‘master_spider.spiders’]
NEWSPIDER_MODULE = ‘master_spider.spiders’

SCHEDULER = "scrapy_redis.scheduler.Scheduler"
DUPEFILTER_CLASS = “scrapy_redis.dupefilter.RFPDupeFilter”

SPLASH_URL = ‘http://192.168.126.128:8050

HTTPCACHE_ENABLED = True
HTTPCACHE_STORAGE = ‘scrapy_splash.SplashAwareFSCacheStorage’

ROBOTSTXT_OBEY = False

SPIDER_MIDDLEWARES = {
‘scrapy_splash.SplashDeduplicateArgsMiddleware’: 100,
# ‘master_spider.middlewares.MasterSpiderSpiderMiddleware’: 543,
}

DOWNLOADER_MIDDLEWARES = {
# ‘master_spider.middlewares.MasterSpiderDownloaderMiddleware’: 543,
‘scrapy_splash.SplashCookiesMiddleware’: 723,
‘scrapy_splash.SplashMiddleware’: 725,
‘scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware’: 810,
‘master_spider.middlewares.RandomUserAgentMiddleware’: 410,
‘master_spider.middlewares.RandomDelayMiddleware’: 999,
}

ITEM_PIPELINES = {
# ‘master_spider.pipelines.MasterSpiderPipeline’: 300,
# ‘scrapy_redis.pipelines.RedisPipeline’: 300,
‘master_spider.pipelines.SaveToES’: 310
}

REDIS_HOST = 'localhost’
REDIS_PORT = 6379

DATETIME_FORMAT = “%Y-%m-%d %H:%M:%S”

RANDOM_DELAY = 2

应该也不是spider文件夹中存在好几个爬虫文件的问题,因为删了也没用

写回答

1回答

bobby

2020-04-22

你留下qq 我加你看看

0
4
bobby
回复
慕函数7358036
这个经验很好,这个问题我调研一下看看
2020-04-23
共4条回复

Scrapy打造搜索引擎 畅销4年的Python分布式爬虫课

带你彻底掌握Scrapy,用Django+Elasticsearch搭建搜索引擎

5796 学习 · 6290 问题

查看课程