服务器断电,重启后报错

来源:1-4 -OOTB镜像文件使用介绍

慕运维7479159

2018-10-29

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/hadoop/app/tmp/nm-local-dir/usercache/root/filecache/12/__spark_libs__8486989979913789473.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/app/hadoop-2.6.0-cdh5.7.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/10/29 08:52:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
18/10/29 08:52:41 WARN hdfs.BlockReaderFactory: I/O error constructing remote block reader.
java.io.IOException: Got error for OP_READ_BLOCK, status=ERROR, self=/192.168.110.38:38297, remote=/192.168.110.38:50010, for file /computer_log_database_checkpoint1/commits/1904, for pool BP-1471073588-192.168.110.38-1536373434566 block 1075890883_2150062 使用 spark 写入 hdfs 一分钟写入一次, 结果就重启后就出现这个问题

18/10/29 08:52:58 ERROR yarn.ApplicationMaster: User class threw exception: org.apache.spark.sql.streaming.StreamingQueryException: Could not obtain block: BP-1471073588-192.168.110.38-1536373434566:blk_1075890883_2150062 file=/computer_log_database_checkpoint1/commits/1904
=== Streaming Query ===
Identifier: [id = 86d09ddf-57c7-4c0f-9955-7c771433f7b6, runId = 0dd7680b-f326-436f-b5bf-dabce9239e86]
Current Committed Offsets: {KafkaSource[Subscribe[imooc_pk_offset]]: {“imooc_pk_offset”:{“0”:1517102231}}}
Current Available Offsets: {KafkaSource[Subscribe[imooc_pk_offset]]: {“imooc_pk_offset”:{“0”:1517117072}}}

Current State: ACTIVE
Thread State: RUNNABLE

Logical Plan:
Project [info#21.pc AS pc#23, info#21.sj AS sj#24L, info#21.rj AS rj#25, info#21.n AS n#26, info#21.ut AS ut#27, info#21.id AS id#28]
± Project [jsontostructs(StructField(pc,StringType,false), StructField(sj,LongType,false), StructField(rj,StringType,false), StructField(n,StringType,false), StructField(st,LongType,false), StructField(ut,DoubleType,false), StructField(tt,DoubleType,false), StructField(id,IntegerType,false), cast(value#8 as string), Some(Asia/Shanghai), true) AS info#21]
± StreamingExecutionRelation KafkaSource[Subscribe[imooc_pk_offset]], [key#7, value#8, topic#9, partition#10, offset#11L, timestamp#12, timestampType#13]

org.apache.spark.sql.streaming.StreamingQueryException: Could not obtain block: BP-1471073588-192.168.110.38-1536373434566:blk_1075890883_2150062 file=/computer_log_database_checkpoint1/commits/1904
=== Streaming Query ===
Identifier: [id = 86d09ddf-57c7-4c0f-9955-7c771433f7b6, runId = 0dd7680b-f326-436f-b5bf-dabce9239e86]
Current Committed Offsets: {KafkaSource[Subscribe[imooc_pk_offset]]: {“imooc_pk_offset”:{“0”:1517102231}}}
Current Available Offsets: {KafkaSource[Subscribe[imooc_pk_offset]]: {“imooc_pk_offset”:{“0”:1517117072}}}

Current State: ACTIVE
Thread State: RUNNABLE

写回答

1回答

Michael_PK

2018-10-29

blk丢失了,你确定下你现在HDFS是在save mode不,要等退出安全模式才行。你们生产是用streaming读取HDFS数据?

0
3
慕运维7479159
回复
Michael_PK
好的,谢谢
2018-10-29
共3条回复

以慕课网日志分析为例 进入大数据Spark SQL的世界

快速转型大数据:Hadoop,Hive,SparkSQL步步为赢

1644 学习 · 1129 问题

查看课程