question How to send data stored in Kafka from beginning to a new consumer using NiFi? in Archives of Support Questions (Read Only)

How to send data stored in Kafka from beginning to a new consumer using NiFi?

mayank_rathi — Mon, 07 Aug 2017 20:55:07 GMT

Hello All,

How can I add a new NiFi consumer to a Kafka topic and send data from beginning? I do not see any setting in ConsumerKafkaRecord_0_10 processor to accomplish this.

I am looking for functionality similar to --from-beginning parameter in Kafka Console Consumer.

"./bin/kafka-console-consumer --zookeeper zk1:2181,zk2:2181,zk3:2181/kafka --topic LCL_LANG_DSTRBTN1 --from-beginning"

Thanks!!

Re: How to send data stored in Kafka from beginning to a new consumer using NiFi?

bbende — Mon, 07 Aug 2017 22:09:01 GMT

There is a property called "Offset Reset" that you can set to "earliest".

This only takes effect the first time the consumer group consumes from the topic, so if you have already run the processor before then you will need to also change the group id.

Re: How to send data stored in Kafka from beginning to a new consumer using NiFi?

mayank_rathi — Tue, 08 Aug 2017 01:34:45 GMT

Thanks Bryan !!