运行完一个批次之后就直接报错

来源:9-10 -Pull方式整合之本地环境联调

qq_windybeyondw_0

2019-02-28

运行完一个批次之后就直接报错
ERROR scheduler.ReceiverTracker: Deregistered receiver for stream 0:
这个是啥原因,下面直接就报了连接不上,flush测试pull的时候

19/02/28 14:04:40 INFO scheduler.JobScheduler: Finished job streaming job 1551333880000 ms.0 from job set of time 1551333880000 ms
19/02/28 14:04:40 INFO scheduler.JobScheduler: Total delay: 0.400 s for time 1551333880000 ms (execution: 0.336 s)
19/02/28 14:04:40 INFO scheduler.ReceivedBlockTracker: Deleting batches:
19/02/28 14:04:40 INFO scheduler.InputInfoTracker: remove old batch metadata:
19/02/28 14:04:40 INFO receiver.ReceiverSupervisorImpl: Stopping receiver with message: Error starting receiver 0: java.io.IOException: Error connecting to /192.168.137.200:41414
19/02/28 14:04:40 INFO flume.FlumePollingReceiver: Shutting down Flume Polling Receiver
19/02/28 14:04:40 INFO receiver.ReceiverSupervisorImpl: Called receiver onStop
19/02/28 14:04:40 INFO receiver.ReceiverSupervisorImpl: Deregistering receiver 0
19/02/28 14:04:40 ERROR scheduler.ReceiverTracker: Deregistered receiver for stream 0: Error starting receiver 0 - java.io.IOException: Error connecting to /192.168.137.200:41414
at org.apache.avro.ipc.NettyTransceiver.getChannel(NettyTransceiver.java:261)
at org.apache.avro.ipc.NettyTransceiver.(NettyTransceiver.java:203)
at org.apache.avro.ipc.NettyTransceiver.(NettyTransceiver.java:138)
at org.apache.spark.streaming.flume.FlumePollingReceiverKaTeX parse error: $ within math modeanonfun$onStart1.apply(FlumePollingInputDStream.scala:82)atscala.collection.immutable.List.foreach(List.scala:381)atorg.apache.spark.streaming.flume.FlumePollingReceiver.onStart(FlumePollingInputDStream.scala:82)atorg.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:149)atorg.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:131)atorg.apache.spark.streaming.scheduler.ReceiverTracker1.apply(FlumePollingInputDStream.scala:82) at scala.collection.immutable.List.foreach(List.scala:381) at org.apache.spark.streaming.flume.FlumePollingReceiver.onStart(FlumePollingInputDStream.scala:82) at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:149) at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:131) at org.apache.spark.streaming.scheduler.ReceiverTracker1.apply(FlumePollingInputDStream.scala:82)atscala.collection.immutable.List.foreach(List.scala:381)atorg.apache.spark.streaming.flume.FlumePollingReceiver.onStart(FlumePollingInputDStream.scala:82)atorg.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:149)atorg.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:131)atorg.apache.spark.streaming.scheduler.ReceiverTrackerReceiverTrackerEndpointKaTeX parse error: $ within math modeanonfun9.apply(ReceiverTracker.scala:597)atorg.apache.spark.SparkContext9.apply(ReceiverTracker.scala:597) at org.apache.spark.SparkContext9.apply(ReceiverTracker.scala:597)atorg.apache.spark.SparkContext$anonfun34.apply(SparkContext.scala:2173)atorg.apache.spark.SparkContext34.apply(SparkContext.scala:2173) at org.apache.spark.SparkContext34.apply(SparkContext.scala:2173)atorg.apache.spark.SparkContext$anonfun34.apply(SparkContext.scala:2173)atorg.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)atorg.apache.spark.scheduler.Task.run(Task.scala:108)atorg.apache.spark.executor.Executor34.apply(SparkContext.scala:2173) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:108) at org.apache.spark.executor.Executor34.apply(SparkContext.scala:2173)atorg.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)atorg.apache.spark.scheduler.Task.run(Task.scala:108)atorg.apache.spark.executor.ExecutorTaskRunner.run(Executor.scala:335)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused: no further information: /192.168.137.200:41414

写回答

1回答

Michael_PK

2019-02-28

你机器的网络是否正常?这种连不上两个可能:一是网络,而是端口是否了

0
2
Michael_PK
回复
qq_windybeyondw_0
我现在记不清哪个先启动,但是视频里详细说明了必须要先启动哪个,否则就挂了。你仔细听下视频的讲解和操作。这个顺序是有严格要求的
2019-02-28
共2条回复

Spark Streaming实时流处理项目实战

Flume+Kafka+Spark Streaming 构建通用实时流处理平台

1404 学习 · 571 问题

查看课程