没有数据保存,也没有报错,反正就是插入不到数据库中
来源:6-19 保存数据到mysql中 -1
Smile苦笑00
2017-11-20
方法代码如下:
def get_insert_sql(self):
#插入知乎question表的sql语句
insert_sql = """
insert into zhihu_question(
zhihu_id,
topics,
url,
title,
content,
answer_num,
comments_num,
watch_user_num,
click_num,
crawl_time,
)
VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)
"""
zhihu_id = self["zhihu_id"][0]
topics = ",".join(self["topics"])
url = self["url"][0]
title = "".join(self["title"])
content = "".join(self["content"])
print(content)
answer_num = extract_num("".join(self["answer_num"]))
comments_num = extract_num("".join(self["comments_num"]))
if len(self["watch_user_num"]) ==2:
watch_user_num = int(self["watch_user_num"][0])
click_num = int(self["watch_user_num"][1])
else:
watch_user_num = int(self["watch_user_num"][0])
click_num = 0
crawl_time = datetime.datetime.now().strftime(SQL_DATETIME_FORMAT)
params = (
zhihu_id,
topics,
url,
title,
content,
answer_num,
comments_num,
watch_user_num,
click_num,
crawl_time,
)
return insert_sql, params
调试时各个参数都有值
执行的代码
def handle_error(self,failure,item,spider):
#处理异步插入的异常
print (failure,item)
def do_insert(self,cursor,item):
#执行具体的插入语句
#根据不同的item,构建不同的sql,并插入到item中
insert_sql,params = item.get_insert_sql()
print(insert_sql,params)
cursor.execute(insert_sql, params)
他就是不报错也不插入到数据表中.回答已经插入成功
1回答
-
bobby
2017-11-21
之前有个同学也是这个问题, 他是 因为每个item的数据的主键都是一样的造成了数据不停的重复更新同一个记录, 你需要调试一下看看你的代码是否每个item的主键都是一样的
022017-11-23
相似问题