执行日志清理时退出
来源:9-25 -数据清洗作业运行到YARN上
慕粉2110073833
2017-11-10
老师下面是,打出的错误是指信息,是由于资源不够吗? 使用的是阿里云的服务器2核4G的, hadoop后台显示的Memory Total是8G
17/11/09 23:57:33 ERROR scheduler.TaskSetManager: Task 0 in stage 1.0 failed 4 times; aborting job
17/11/09 23:57:33 INFO cluster.YarnScheduler: Removed TaskSet 1.0, whose tasks have all completed, from pool
17/11/09 23:57:33 INFO storage.BlockManagerMaster: Removal of executor 4 requested
17/11/09 23:57:33 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asked to remove non-existent executor 4
17/11/09 23:57:33 INFO storage.BlockManagerMasterEndpoint: Trying to remove executor 4 from BlockManagerMaster.
17/11/09 23:57:33 INFO cluster.YarnScheduler: Cancelling stage 1
17/11/09 23:57:33 INFO scheduler.DAGScheduler: ResultStage 1 (save at SparkStatCleanJobYARN.scala:29) failed in 82.394 s due to Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 4, hadoop001, executor 4): ExecutorLostFailure (executor 4 exited caused by one of the running tasks) Reason: Container marked as failed: container_1510122812790_0009_02_000002 on host: hadoop001. Exit status: 1. Diagnostics: Exception from container-launch.
Container id: container_1510122812790_0009_02_000002
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:561)
at org.apache.hadoop.util.Shell.run(Shell.java:478)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:738)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Container exited with a non-zero exit code 1
1回答
-
Michael_PK
2017-11-10
从这日志看应该是yarn有问题,你到失败作业的yarn页面上,仔细查看下各个链接的日志信息
022017-11-10
相似问题