flume的配置后kafka没有数据

来源:6-5 【实操手册】环境搭建文档 ( 完善版 )

宝慕林8341667

2020-06-04

按照您的flume的配置
nginx.sources=source_1
nginx.channels=channel_1
nginx.sinks=sink_1
nginx.sources.source_1.type=exec
nginx.sources.source_1.command=tail -F /usr/local/nginx/nginx/logs/access.log
nginx.sources.source_1.channels=channel_1
nginx.sinks.sink_1.channel=channel_1
nginx.channels.channel_1.type=org.apache.flume.channel.kafka.KafkaChannel
nginx.channels.channel_1.kafka.bootstrap.servers=iotsca1:9092,iotsca2:9092,iotsca3:9092
nginx.channels.channel_1.kafka.topic=centos1_flume_channel
nginx.channels.channel_1.kafka.consumer.group.id=centos1_flume_channel_consumer
nginx.channels.channel_1.kafka.consumer.timeout.ms=70000
nginx.channels.channel_1.kafka.consumer.request.timeout.ms=80000
nginx.channels.channel_1.kafka.consumer.fetch.max.wait.ms=7000
nginx.channels.channel_1.kafka.consumer.offset.flush.interval.ms=50000
nginx.channels.channel_1.kafka.consumer.session.timeout.ms=70000
nginx.channels.channel_1.kafka.consumer.heartbeat.interval.ms=60000
nginx.channels.channel_1.kafka.consumer.enable.auto.commit=false
nginx.sinks.sink_1.type=avro
nginx.sinks.sink_1.hostname=iotsca3
nginx.sinks.sink_1.port=52020
我的kafka没有收到数据
这个是flume的log
2020-06-04 14:57:50,421 INFO org.apache.flume.conf.FlumeConfiguration: Processing:channel_1
2020-06-04 14:57:50,421 INFO org.apache.flume.conf.FlumeConfiguration: Processing:channel_1
2020-06-04 14:57:50,421 INFO org.apache.flume.conf.FlumeConfiguration: Processing:channel_1
2020-06-04 14:57:50,421 WARN org.apache.flume.conf.FlumeConfiguration: Agent configuration for ‘nginx’ has no configfilters.
2020-06-04 14:57:50,454 INFO org.apache.flume.conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [nginx]
2020-06-04 14:57:50,454 WARN org.apache.flume.node.AbstractConfigurationProvider: No configuration found for this host:tier1
2020-06-04 14:57:50,464 INFO org.apache.flume.node.Application: Starting new configuration:{ sourceRunners:{} sinkRunners:{} channels:{} }
2020-06-04 14:57:50,543 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2020-06-04 14:57:50,626 INFO org.mortbay.log: jetty-6.1.26.cloudera.4
2020-06-04 14:57:50,651 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:41414

写回答

1回答

小简同学

2020-06-04

同学你好,你的flume配置可能有错,根据你的flume日志,显示了两个错误,1. nginx这个名称没有配置, 你检查下flume的启动命令是否有把配置的名称设为nginx 2. tire1这个host没有定义,这个你也检查一下。
祝学习愉快!

0
1
宝慕林8341667
谢谢老师
2020-06-05
共1条回复

Spark2.x+协同过滤算法,开发企业级个性化推荐系统

一门既讲算法原理又有完整的推荐系统架构的全能课

330 学习 · 91 问题

查看课程