[hadoop@hadoop001 spark-2.1.0-bin-2.6.0-cdh5.7.0]$ export HADOOP_CONF_DIR=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop[hadoop@hadoop001 spark-2.1.0-bin-2.6.0-cdh5.7.0]$ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster --executor-memory 1G --num-executors 1 /home/hadoop/app/spark-2.1.0-bin-2.6.0-cdh5.7.0/examples/jars/spark-examples_2.11-2.1.0.jar 4

Warning: Master yarn-cluster is deprecated since 2.0. Please use master "yarn" with specified deploy mode instead.

17/12/22 22:42:00 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

17/12/22 22:42:01 INFO RMProxy: Connecting to ResourceManager at /0.0.0.0:8032

17/12/22 22:42:01 INFO Client: Requesting a new application from cluster with 1 NodeManagers

17/12/22 22:42:01 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container)

17/12/22 22:42:01 INFO Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead

17/12/22 22:42:01 INFO Client: Setting up container launch context for our AM

17/12/22 22:42:01 INFO Client: Setting up the launch environment for our AM container

17/12/22 22:42:01 INFO Client: Preparing resources for our AM container

17/12/22 22:42:04 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.

17/12/22 22:42:09 INFO Client: Uploading resource file:/tmp/spark-9e737c24-aaf4-45b6-a22b-4f6e96f7c6d4/__spark_libs__4278337255131778568.zip -> hdfs://hadoop001:8020/user/hadoop/.sparkStaging/application_1514011258951_0001/__spark_libs__4278337255131778568.zip

17/12/22 22:42:11 INFO Client: Uploading resource file:/home/hadoop/app/spark-2.1.0-bin-2.6.0-cdh5.7.0/examples/jars/spark-examples_2.11-2.1.0.jar -> hdfs://hadoop001:8020/user/hadoop/.sparkStaging/application_1514011258951_0001/spark-examples_2.11-2.1.0.jar

17/12/22 22:42:11 INFO Client: Uploading resource file:/tmp/spark-9e737c24-aaf4-45b6-a22b-4f6e96f7c6d4/__spark_conf__46156238239326766.zip -> hdfs://hadoop001:8020/user/hadoop/.sparkStaging/application_1514011258951_0001/__spark_conf__.zip

17/12/22 22:42:11 INFO SecurityManager: Changing view acls to: hadoop

17/12/22 22:42:11 INFO SecurityManager: Changing modify acls to: hadoop

17/12/22 22:42:11 INFO SecurityManager: Changing view acls groups to: 

17/12/22 22:42:11 INFO SecurityManager: Changing modify acls groups to: 

17/12/22 22:42:11 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hadoop); groups with view permissions: Set(); users  with modify permissions: Set(hadoop); groups with modify permissions: Set()

17/12/22 22:42:11 INFO Client: Submitting application application_1514011258951_0001 to ResourceManager

17/12/22 22:42:12 INFO YarnClientImpl: Submitted application application_1514011258951_0001

17/12/22 22:42:13 INFO Client: Application report for application_1514011258951_0001 (state: ACCEPTED)

17/12/22 22:42:13 INFO Client: 

client token: N/A

diagnostics: N/A

ApplicationMaster host: N/A

ApplicationMaster RPC port: -1

queue: root.hadoop

start time: 1514011332174

final status: UNDEFINED

tracking URL: http://hadoop001:8088/proxy/application_1514011258951_0001/

user: hadoop

17/12/22 22:42:14 INFO Client: Application report for application_1514011258951_0001 (state: ACCEPTED)

17/12/22 22:42:15 INFO Client: Application report for application_1514011258951_0001 (state: ACCEPTED)

17/12/22 22:42:16 INFO Client: Application report for application_1514011258951_0001 (state: ACCEPTED)

17/12/22 22:42:17 INFO Client: Application report for application_1514011258951_0001 (state: ACCEPTED)

17/12/22 22:42:18 INFO Client: Application report for application_1514011258951_0001 (state: ACCEPTED)

17/12/22 22:42:19 INFO Client: Application report for application_1514011258951_0001 (state: ACCEPTED)

17/12/22 22:42:20 INFO Client: Application report for application_1514011258951_0001 (state: ACCEPTED)

17/12/22 22:42:21 INFO Client: Application report for application_1514011258951_0001 (state: ACCEPTED)

17/12/22 22:42:22 INFO Client: Application report for application_1514011258951_0001 (state: ACCEPTED)

17/12/22 22:42:23 INFO Client: Application report for application_1514011258951_0001 (state: RUNNING)

17/12/22 22:42:23 INFO Client: 

client token: N/A

diagnostics: N/A

ApplicationMaster host: 192.168.140.128

ApplicationMaster RPC port: 0

queue: root.hadoop

start time: 1514011332174

final status: UNDEFINED

tracking URL: http://hadoop001:8088/proxy/application_1514011258951_0001/

user: hadoop

17/12/22 22:42:24 INFO Client: Application report for application_1514011258951_0001 (state: RUNNING)

17/12/22 22:42:25 INFO Client: Application report for application_1514011258951_0001 (state: RUNNING)

17/12/22 22:42:26 INFO Client: Application report for application_1514011258951_0001 (state: RUNNING)

17/12/22 22:42:27 INFO Client: Application report for application_1514011258951_0001 (state: RUNNING)

17/12/22 22:42:28 INFO Client: Application report for application_1514011258951_0001 (state: RUNNING)

17/12/22 22:42:29 INFO Client: Application report for application_1514011258951_0001 (state: RUNNING)

17/12/22 22:42:30 INFO Client: Application report for application_1514011258951_0001 (state: FINISHED)

17/12/22 22:42:30 INFO Client: 

client token: N/A

diagnostics: N/A

ApplicationMaster host: 192.168.140.128

ApplicationMaster RPC port: 0

queue: root.hadoop

start time: 1514011332174

final status: SUCCEEDED

tracking URL: http://hadoop001:8088/proxy/application_1514011258951_0001/A

user: hadoop

17/12/22 22:42:30 INFO ShutdownHookManager: Shutdown hook called

17/12/22 22:42:30 INFO ShutdownHookManager: Deleting directory /tmp/spark-9e737c24-aaf4-45b6-a22b-4f6e96f7c6d4