报错信息
来源:9-24 -Spark on YARN基础

慕运维7479159
2017-12-23
[hadoop@hadoop001 spark-2.1.0-bin-2.6.0-cdh5.7.0]$ export HADOOP_CONF_DIR=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop[hadoop@hadoop001 spark-2.1.0-bin-2.6.0-cdh5.7.0]$ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster --executor-memory 1G --num-executors 1 /home/hadoop/app/spark-2.1.0-bin-2.6.0-cdh5.7.0/examples/jars/spark-examples_2.11-2.1.0.jar 4
Warning: Master yarn-cluster is deprecated since 2.0. Please use master "yarn" with specified deploy mode instead.
17/12/22 22:42:00 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/12/22 22:42:01 INFO RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
17/12/22 22:42:01 INFO Client: Requesting a new application from cluster with 1 NodeManagers
17/12/22 22:42:01 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)
17/12/22 22:42:01 INFO Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
17/12/22 22:42:01 INFO Client: Setting up container launch context for our AM
17/12/22 22:42:01 INFO Client: Setting up the launch environment for our AM container
17/12/22 22:42:01 INFO Client: Preparing resources for our AM container
17/12/22 22:42:04 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
17/12/22 22:42:09 INFO Client: Uploading resource file:/tmp/spark-9e737c24-aaf4-45b6-a22b-4f6e96f7c6d4/__spark_libs__4278337255131778568.zip -> hdfs://hadoop001:8020/user/hadoop/.sparkStaging/application_1514011258951_0001/__spark_libs__4278337255131778568.zip
17/12/22 22:42:11 INFO Client: Uploading resource file:/home/hadoop/app/spark-2.1.0-bin-2.6.0-cdh5.7.0/examples/jars/spark-examples_2.11-2.1.0.jar -> hdfs://hadoop001:8020/user/hadoop/.sparkStaging/application_1514011258951_0001/spark-examples_2.11-2.1.0.jar
17/12/22 22:42:11 INFO Client: Uploading resource file:/tmp/spark-9e737c24-aaf4-45b6-a22b-4f6e96f7c6d4/__spark_conf__46156238239326766.zip -> hdfs://hadoop001:8020/user/hadoop/.sparkStaging/application_1514011258951_0001/__spark_conf__.zip
17/12/22 22:42:11 INFO SecurityManager: Changing view acls to: hadoop
17/12/22 22:42:11 INFO SecurityManager: Changing modify acls to: hadoop
17/12/22 22:42:11 INFO SecurityManager: Changing view acls groups to:
17/12/22 22:42:11 INFO SecurityManager: Changing modify acls groups to:
17/12/22 22:42:11 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); groups with view permissions: Set(); users with modify permissions: Set(hadoop); groups with modify permissions: Set()
17/12/22 22:42:11 INFO Client: Submitting application application_1514011258951_0001 to ResourceManager
17/12/22 22:42:12 INFO YarnClientImpl: Submitted application application_1514011258951_0001
17/12/22 22:42:13 INFO Client: Application report for application_1514011258951_0001 (state: ACCEPTED)
17/12/22 22:42:13 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.hadoop
start time: 1514011332174
final status: UNDEFINED
tracking URL: http://hadoop001:8088/proxy/application_1514011258951_0001/
user: hadoop
17/12/22 22:42:14 INFO Client: Application report for application_1514011258951_0001 (state: ACCEPTED)
17/12/22 22:42:15 INFO Client: Application report for application_1514011258951_0001 (state: ACCEPTED)
17/12/22 22:42:16 INFO Client: Application report for application_1514011258951_0001 (state: ACCEPTED)
17/12/22 22:42:17 INFO Client: Application report for application_1514011258951_0001 (state: ACCEPTED)
17/12/22 22:42:18 INFO Client: Application report for application_1514011258951_0001 (state: ACCEPTED)
17/12/22 22:42:19 INFO Client: Application report for application_1514011258951_0001 (state: ACCEPTED)
17/12/22 22:42:20 INFO Client: Application report for application_1514011258951_0001 (state: ACCEPTED)
17/12/22 22:42:21 INFO Client: Application report for application_1514011258951_0001 (state: ACCEPTED)
17/12/22 22:42:22 INFO Client: Application report for application_1514011258951_0001 (state: ACCEPTED)
17/12/22 22:42:23 INFO Client: Application report for application_1514011258951_0001 (state: RUNNING)
17/12/22 22:42:23 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 192.168.140.128
ApplicationMaster RPC port: 0
queue: root.hadoop
start time: 1514011332174
final status: UNDEFINED
tracking URL: http://hadoop001:8088/proxy/application_1514011258951_0001/
user: hadoop
17/12/22 22:42:24 INFO Client: Application report for application_1514011258951_0001 (state: RUNNING)
17/12/22 22:42:25 INFO Client: Application report for application_1514011258951_0001 (state: RUNNING)
17/12/22 22:42:26 INFO Client: Application report for application_1514011258951_0001 (state: RUNNING)
17/12/22 22:42:27 INFO Client: Application report for application_1514011258951_0001 (state: RUNNING)
17/12/22 22:42:28 INFO Client: Application report for application_1514011258951_0001 (state: RUNNING)
17/12/22 22:42:29 INFO Client: Application report for application_1514011258951_0001 (state: RUNNING)
17/12/22 22:42:30 INFO Client: Application report for application_1514011258951_0001 (state: FINISHED)
17/12/22 22:42:30 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 192.168.140.128
ApplicationMaster RPC port: 0
queue: root.hadoop
start time: 1514011332174
final status: SUCCEEDED
tracking URL: http://hadoop001:8088/proxy/application_1514011258951_0001/A
user: hadoop
17/12/22 22:42:30 INFO ShutdownHookManager: Shutdown hook called
17/12/22 22:42:30 INFO ShutdownHookManager: Deleting directory /tmp/spark-9e737c24-aaf4-45b6-a22b-4f6e96f7c6d4
1回答
-
Michael_PK
2017-12-23
报错信息在哪?没见到error 啊
00
相似问题