Flume配置文件报错

来源:9-10 -Pull方式整合之本地环境联调

Clay_mu

2019-05-30

老师您好,我在本地联调的时候Flume是正常启动的,但是在SparkStreaming应用程序启动以后Flume就会报错。
错误信息:

2019-05-30 00:27:28,705 (Spark Sink Processor Thread - 10) [WARN - org.apache.spark.streaming.flume.sink.Logging$class.logWarning(Logging.scala:80)] Error while processing transaction.
java.lang.IllegalStateException: begin() called when transaction is OPEN!
	at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
	at org.apache.flume.channel.BasicTransactionSemantics.begin(BasicTransactionSemantics.java:131)
	at org.apache.spark.streaming.flume.sink.TransactionProcessor$$anonfun$populateEvents$1.apply(TransactionProcessor.scala:114)
	at org.apache.spark.streaming.flume.sink.TransactionProcessor$$anonfun$populateEvents$1.apply(TransactionProcessor.scala:113)
	at scala.Option.foreach(Option.scala:236)
	at org.apache.spark.streaming.flume.sink.TransactionProcessor.populateEvents(TransactionProcessor.scala:113)
	at org.apache.spark.streaming.flume.sink.TransactionProcessor.call(TransactionProcessor.scala:243)
	at org.apache.spark.streaming.flume.sink.TransactionProcessor.call(TransactionProcessor.scala:43)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2019-05-30 00:27:28,706 (Spark Sink Processor Thread - 10) [WARN - org.apache.spark.streaming.flume.sink.Logging$class.logWarning(Logging.scala:59)] Spark was unable to successfully process the events. Transaction is being rolled back.
2019-05-30 00:27:28,706 (New I/O  worker #1) [WARN - org.apache.spark.streaming.flume.sink.Logging$class.logWarning(Logging.scala:59)] Received an error batch - no events were received from channel! 

我使用的是2.3.3版本的Spark,但是maven的依赖中没有2.3.3的依赖,所以用的是2.3.2的依赖,在Push style中是可以运行的。会不会是因为Spark的版本问题啊?

写回答

2回答

Michael_PK

2019-05-30

这些都是本地一起测试是吗?启动顺序OK的不?

1
4
Clay_mu
老师,问题解决了!!! https://stackoverflow.com/questions/34657391/unable-to-pull-events-in-spark-streaming-application-from-flume 参考的是这个链接里的解决方案。根据第三个回答,我去Flume的lib文件中看了一下,lib文件中的scala-library jar包的版本是2.10.1,我们课程里使用的是2.11.x的scala。所以我先把2.10.1的jar包删掉,然后下载对应scala版本的jar包放进去就解决掉这个问题了。
2019-05-31
共4条回复

Michael_PK

2019-05-30

看样子还是对接的sink配置有点问题

1
1
Clay_mu
老师,问题解决了!!! https://stackoverflow.com/questions/34657391/unable-to-pull-events-in-spark-streaming-application-from-flume 参考的是这个链接里的解决方案。根据第三个回答,我去Flume的lib文件中看了一下,lib文件中的scala-library jar包的版本是2.10.1,我们课程里使用的是2.11.x的scala。所以我先把2.10.1的jar包删掉,然后下载对应scala版本的jar包放进去就解决掉这个问题了。
2019-06-01
共1条回复

Spark Streaming实时流处理项目实战

Flume+Kafka+Spark Streaming 构建通用实时流处理平台

1404 学习 · 571 问题

查看课程