We have few flume agents which have kafka as channel and hdfs as sink. Recently, we added few more flume agents and all the sources have re-balanced and have detected new flume agents however the channel topics consumer group are not re-balancing with the new flume agents.
Below is my channel configuration.
TAAS.channels.taasChnl2.type = org.apache.flume.channel.kafka.KafkaChannel TAAS.channels.taasChnl2.brokerList = kafka1.test.com:6667 TAAS.channels.taasChnl2.zookeeperConnect = server1.test.com:2181,server2.test.com:2181,server3.test.com:2181/kafka1 TAAS.channels.taasChnl2.topic = CHNL_LOG_TRANSFORM TAAS.channels.taasChnl2.groupId = TAAS_FLUME_P_1 TAAS.channels.taasChnl2.parseAsFlumeEvent = true
The above was originally posted in the Community Help Track. On Tue Jun 4 20:09 UTC 2019, a member of the HCC moderation staff moved it to the Data Ingestion & Streaming track. The Community Help Track is intended for questions about using the HCC site itself.