flume 链接报错

来源：3-8 -Flume实战案例三(重点掌握)

starkpan

2019-01-14

老师，我想把一个机器上的日志抓取到另一台机器，但是两台链接不成功。
从一台机器监听一个文件，将新产生的文件，传输到另一个机器
agent选型：
exec-source +memory channel + avro-sink
avro source +memory channel + hdfs sink

第一个配置文件
#exec-memory-avro.conf

spider001.sources = exec-source
spider001.sinks = avro-sink
spider001.channels = memory-channel

spider001.sources.exec-source.type = exec
spider001.sources.exec-source.command=tail -f /home/hadoop/data/liepin_2019-01-08.json
spider001.sources.exec-source.shell = /bin/sh -c

spider001.sinks.avro-sink.type = avro
spider001.sinks.avro-sink.hostname = 192.168.1.105
spider001.sinks.avro-sink.port = 41414

spider001.channels.memory-channel.type = memory

spider001.sources.exec-source.channels = memory-channel
spider001.sinks.avro-sink.channel = memory-channel

第二个配置文件
#avro-memory-hdfs.conf

hadoopOne.sources = spider001-source
hadoopOne.sinks = hdfs-sink
hadoopOne.channels = memory-channel

hadoopOne.sources.spider001-source.type = avro
hadoopOne.sources.spider001-source.bind=192.168.1.106
hadoopOne.sources.spider001-source.port = 41414

hadoopOne.sinks.hdfs-sink.type = hdfs
hadoopOne.sinks.hdfs-sink.hdfs.path =hdfs://192.168.1.105:9000/flume
hadoopOne.sinks.hdfs-sink.hdfs.rollInterval= 0
hadoopOne.sinks.hdfs-sink.hdfs.rollSize = 131072
hadoopOne.sinks.hdfs-sink.hdfs.fileType=DataStream

hadoopOne.channels.memory-channel.type = memory

hadoopOne.sources.spider001-source.channels = memory-channel
hadoopOne.sinks.hdfs-sink.channel = memory-channel

启动
flume-ng agent -n hadoopOne --conf $FLUM_HOME/conf -f $FLUM_HOME/conf/hadoopOne.conf -Dflume.root.logger=INFO,console

flume-ng agent -n spider001 --conf $FLUM_HOME/conf -f $FLUM_HOME/conf/spider001.conf -Dflume.root.logger=INFO,console

启动hadoopOne 的，会报错无法注册
org.jboss.netty.channel.ChannelException: Failed to bind to: /192.168.1.106:41414
Caused by: java.net.BindException: Cannot assign requested address

启动spider001（数据源）
2019-01-14 09:14:36,021 (lifecycleSupervisor-1-2) [WARN - org.apache.flume.sink.AbstractRpcSink.start(AbstractRpcSink.java:294)] Unable to create Rpc client using hostname: 192.168.1.107, port: 41414
org.apache.flume.FlumeException: NettyAvroRpcClient { host: 192.168.1.105, port: 41414 }: RPC connection error
Caused by: java.io.IOException: Error connecting to /192.168.1.105:41414

写回答

2回答

Michael_PK

2019-01-15

已采纳

1）如果exec-memory-avro.conf是A机器，avro-memory-hdfs.conf是B机器，一定要先在B机器上先启动agent，然后在A机器上启动agent，这个顺序不能错，我看了下你的agent配置，并没发现明显的错误信息。

2）你的日志里为什么有192.168.1.107这台机器？你配置的两个agent分别是192.168.1.105和192.168.1.106，这里检查下是什么原因？

starkpan

1、谢谢老师。启动顺序没错。先启动的接收端，再启动的收集端。配置文件没啥错的话，只能找找其他原因了。 2、IP出现107，是因为105和106怎么试都有问起，于是又起了一台虚拟机，并修改文件进行链接，结果还是一样的错。提问的时候，报错信息是在107和105粘的，配置文件是事先写好的笔记。为了保证报错和笔记显示一致，提问时，手动修改了报错信息IP，漏掉了一个107没改。

2019-01-15

共1条回复

Michael_PK

2019-01-14

两台机器的agent启动顺序对不

starkpan

各种顺序都试了

2019-01-14

共1条回复

Spark Streaming实时流处理项目实战

Flume+Kafka+Spark Streaming 构建通用实时流处理平台

1404 学习 · 571 问题