使用spark-submit时报以下错误

来源:9-6 -Push方式整合之服务器环境联调

96年的nash

2019-11-05

spark-submit \
--class com.imooc.spark.FlumePushWordCount \
--master local[2] \
--packages org.apache.spark:spark-streaming-flume_2.11:2.2.0 \
/home/hadoop/lib/sparktrain-1.0.jar \
hadoop000 41414
java.lang.NoClassDefFoundError: org/apache/flume/source/avro/AvroSourceProtocol
	at org.apache.spark.streaming.flume.FlumeReceiver.responder$lzycompute(FlumeInputDStream.scala:146)
	at org.apache.spark.streaming.flume.FlumeReceiver.responder(FlumeInputDStream.scala:145)
	at org.apache.spark.streaming.flume.FlumeReceiver.initServer(FlumeInputDStream.scala:162)
	at org.apache.spark.streaming.flume.FlumeReceiver.onStart(FlumeInputDStream.scala:169)
	at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:149)
	at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:131)
	at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:607)
	at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverTrackerEndpoint$$anonfun$9.apply(ReceiverTracker.scala:597)
	at org.apache.spark.SparkContext$$anonfun$34.apply(SparkContext.scala:2173)
	at org.apache.spark.SparkContext$$anonfun$34.apply(SparkContext.scala:2173)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:108)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

从头到尾一直循环报这个错误,最后内存一处自动关闭了

写回答

1回答

96年的nash

提问者

2019-11-05

本地环境联调时没毛病,在服务器上就不行了

0
5
Michael_PK
回复
96年的nash
完全可以的呢。本地运行就是学习的正确姿势,一路本地local搞完先
2019-11-05
共5条回复

Spark Streaming实时流处理项目实战

Flume+Kafka+Spark Streaming 构建通用实时流处理平台

1404 学习 · 571 问题

查看课程