spark-submit无法跑在yarn上

来源:6-7 Spark部署及服务器端测试

慕田峪0177977

2020-09-24

[hadoop@hadoop000 spark-3.0.0-bin-2.6.0-cdh5.16.2]$ bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn examples/jars/spark-examples*.jar 2



Exception in thread "main" org.apache.spark.SparkException: Could not load YARN classes. This copy of Spark may not have been compiled with YARN support.

at org.apache.spark.deploy.SparkSubmit.error(SparkSubmit.scala:936)

at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:255)

at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)

at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)

at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)

at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)

at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)

at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)

at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)


写回答

1回答

Michael_PK

2020-09-24

你这个spark是自己编译的还是?

This copy of Spark may not have been compiled with YARN support.

不支持yarn呢

0
2
Michael_PK
回复
慕田峪0177977
你编译的时候有-Pyarn吗
2020-09-24
共2条回复

Spark3实时处理-Streaming+StructuredStreaming实战

实战Spark3实时处理,掌握两套企业级处理方案

340 学习 · 238 问题

查看课程