Support Questions
Find answers, ask questions, and share your expertise

How to send data stored in Kafka from beginning to a new consumer using NiFi?

Solved Go to solution

How to send data stored in Kafka from beginning to a new consumer using NiFi?

Contributor

Hello All,

How can I add a new NiFi consumer to a Kafka topic and send data from beginning? I do not see any setting in ConsumerKafkaRecord_0_10 processor to accomplish this.

I am looking for functionality similar to --from-beginning parameter in Kafka Console Consumer.

"./bin/kafka-console-consumer --zookeeper zk1:2181,zk2:2181,zk3:2181/kafka --topic LCL_LANG_DSTRBTN1 --from-beginning"

Thanks!!

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: How to send data stored in Kafka from beginning to a new consumer using NiFi?

There is a property called "Offset Reset" that you can set to "earliest".

This only takes effect the first time the consumer group consumes from the topic, so if you have already run the processor before then you will need to also change the group id.

View solution in original post

2 REPLIES 2
Highlighted

Re: How to send data stored in Kafka from beginning to a new consumer using NiFi?

There is a property called "Offset Reset" that you can set to "earliest".

This only takes effect the first time the consumer group consumes from the topic, so if you have already run the processor before then you will need to also change the group id.

View solution in original post

Highlighted

Re: How to send data stored in Kafka from beginning to a new consumer using NiFi?

Contributor

Thanks Bryan !!