Kafka sink启动失败

来源:11-5 -使用KafkaSInk将Flume收集到的数据输出到Kafka

Clay_mu

2019-06-18

老师您好,按照视频中的配置,flume总是启动失败。pom文件中的kafka依赖已经注释掉了。Flume和Kafka下的lib文件夹中的jar包都对应上了,但是还是会报下面的错误:

2019-06-18 11:17:19,907 (lifecycleSupervisor-1-3) [ERROR - org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:253)] Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@6f3f9da3 counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.NoClassDefFoundError: org/apache/kafka/common/utils/Utils
	at kafka.utils.CoreUtils$.createObject(CoreUtils.scala:221)
	at kafka.producer.Producer.<init>(Producer.scala:61)
	at kafka.javaapi.producer.Producer.<init>(Producer.scala:26)
	at org.apache.flume.sink.kafka.KafkaSink.start(KafkaSink.java:164)
	at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
	at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
	at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.utils.Utils
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 14 more
2019-06-18 11:17:19,909 (lifecycleSupervisor-1-3) [ERROR - org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:264)] Unsuccessful attempt to shutdown component: {} due to missing dependencies. Please shutdown the agentor disable this component, or the agent will bein an undefined state.
java.lang.NullPointerException
	at org.apache.flume.sink.kafka.KafkaSink.stop(KafkaSink.java:171)
	at org.apache.flume.sink.DefaultSinkProcessor.stop(DefaultSinkProcessor.java:53)
	at org.apache.flume.SinkRunner.stop(SinkRunner.java:115)
	at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:259)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

这个还是jar包冲突的问题么?

写回答

1回答

Michael_PK

2019-06-18

这个看起来是版本的兼容问题,你的版本和上课一样吗?版本不兼容导致里面的依赖的东西有冲突

0
3
Clay_mu
回复
Michael_PK
好的,谢谢老师。我再查一下。换个scala试试
2019-06-18
共3条回复

Spark Streaming实时流处理项目实战

Flume+Kafka+Spark Streaming 构建通用实时流处理平台

1404 学习 · 571 问题

查看课程