https://community.cloudera.com/t5/Support-Questions/Problem-about-Configuring-Flume-as-Kafka-Consumer/m-p/61018#M45285 <P>My Configuration is</P><P>Cloudera Manager&nbsp;5.12.1</P><P>Flume 1.7.0</P><P>Kakfa&nbsp;2.2.0-1.2.2.0.p0.68</P><P>&nbsp;</P><P>I created topic "test" in kafka, and would like to configure flume to act as consumer to fetch data from this topic and save it to HDFS.</P><P>My flume configuration is</P><P>#################################</P><P>tier1.sources = source1<BR />tier1.channels = channel1<BR />tier1.sinks = sink1</P><P>tier1.sources.source1.type = org.apache.flume.source.kafka.KafkaSource<BR />tier1.sources.source1.zookeeperConnect = <SPAN>bda03:2182,&nbsp;bda04:2182,</SPAN>bda05:2182<BR />tier1.sources.source1.topic = test<BR />tier1.sources.source1.groupId = flume<BR />tier1.sources.source1.channels = channel1<BR />tier1.sources.source1.interceptors = i1<BR />tier1.sources.source1.interceptors.i1.type = timestamp<BR />tier1.sources.source1.kafka.consumer.timeout.ms = 100</P><P>tier1.channels.channel1.type = memory<BR />tier1.channels.channel1.capacity = 10000<BR />tier1.channels.channel1.transactionCapacity = 1000</P><P>tier1.sinks.sink1.type = hdfs<BR />tier1.sinks.sink1.hdfs.path = /user/flume/tmp/<BR />tier1.sinks.sink1.hdfs.rollInterval = 5<BR />tier1.sinks.sink1.hdfs.rollSize = 0<BR />tier1.sinks.sink1.hdfs.rollCount = 0<BR />tier1.sinks.sink1.hdfs.fileType = DataStream<BR />tier1.sinks.sink1.channel = channel1</P><P>######################################</P><P>The configuration was just copied from&nbsp;<A href="https://www.cloudera.com/documentation/kafka/latest/topics/kafka_flume.html" target="_blank">https://www.cloudera.com/documentation/kafka/latest/topics/kafka_flume.html</A></P><P>&nbsp;</P><P>After starting the flume agent, there was no error message in flume and kafka log file.</P><P>&nbsp;</P><P>I tried to put some records to topics:</P><P>$ kafka-console-producer --broker-list bda03:9092,bda04:9092,bda05:9092 --topic test</P><P>Hello</P><P>This is new message</P><P>&nbsp;</P><P>However, there was nothing put into HDFS.</P><P>Then, I tried to&nbsp;retreive message from topic "test" via command line (with consumer group flume)</P><P>$&nbsp;kafka-console-consumer --zookeeper bda03:2182,bda04:2182,bda05:2182 --topic test --consumer-property group.id=flume</P><P>The output was:</P><P>Hello</P><P>This is new message</P><P>&nbsp;</P><P>If the flume agent could fetch message from the topic, then the output of the above command should be empty.</P><P>The result indicated that the flume agent could not get any data from Kafka.</P><P>&nbsp;</P><P>Can anyone help?</P><P>Thanks!</P><P>&nbsp;</P><P>&nbsp;</P> Fri, 16 Sep 2022 12:24:58 GMT alfredng 2022-09-16T12:24:58Z