yarn 提交spark

来源：6-6 -从词频统计功能着手入门Spark Streaming

慕九州7382466

2018-11-09

package com.hjx

import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.{SparkConf, SparkContext}

object WordCount {
def main(args: Array[String]): Unit = {
val appName = “WordCount”

val conf = new SparkConf().setAppName(appName)
val sc = new SparkContext(conf)
val ssc = new StreamingContext(sc,Seconds(5))
val lines = ssc.socketTextStream("localhost",9999)
val words = lines.flatMap(_.split(" "))
val result = words.map((_,1)).reduceByKey(_+_)
ssc.checkpoint("hdfs:///test/")
ssc.start()
ssc.awaitTermination()

}
}
以上是代码，

写回答

2回答

Michael_PK

2018-11-09

代码的output在哪

Michael_PK

慕九州7382466

你自己看看result有输出吗

2018-11-09

共5条回复

慕九州7382466

提问者

2018-11-09

ERROR ApplicationMaster: Uncaught exception:
java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]
    at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
    at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
    at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:201)
    at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:498)
    at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:345)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply$mcV$sp(ApplicationMaster.scala:260)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$5.run(ApplicationMaster.scala:815)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
    at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:814)
    at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:259)
    at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:839)
    at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala)

这是日志的错误

Spark Streaming实时流处理项目实战

Flume+Kafka+Spark Streaming 构建通用实时流处理平台

1404 学习 · 571 问题

查看课程

相似问题

spark spark

回答 1

Spark Streaming作业是运行在yarn上吗？

回答 1

使用spark-submit 提交flume作业报错

回答 1

老师，有多个streaming架包作业能同时运行吗？

回答 1

关于在yarn上submit

回答 3

打开慕课网App查看更多内容