Support Questions
Find answers, ask questions, and share your expertise

NiFi - ConsumeKafka(0.10) How to achieve seektobeginning?

Rising Star

I understand that for the very first time, we can use the Offset Reset to earliest would do. However, subsequently if for load testing and validations if the offset needs to be reset to the beginning, how can we achieve that through NiFi?

1 ACCEPTED SOLUTION

Hello Kumar,

There's an ongoing effort to provide a feature to reset Kafka offset from NiFi UI. We're trying to provide a way for NiFi user to reset state, that is managed externally, such as Kafka offset.

https://issues.apache.org/jira/browse/NIFI-2078

https://github.com/apache/nifi/pull/563

Specifically, clear offset by submitting new offset as -1.

https://github.com/apache/nifi/pull/563/files#diff-c473fdb0ebb08dd72a64d0e6536e85c0R396

I hope I'll be able to resume working on that JIRA soon, please watch the JIRA issue to be updated progress.

View solution in original post

3 REPLIES 3

Hello Kumar,

There's an ongoing effort to provide a feature to reset Kafka offset from NiFi UI. We're trying to provide a way for NiFi user to reset state, that is managed externally, such as Kafka offset.

https://issues.apache.org/jira/browse/NIFI-2078

https://github.com/apache/nifi/pull/563

Specifically, clear offset by submitting new offset as -1.

https://github.com/apache/nifi/pull/563/files#diff-c473fdb0ebb08dd72a64d0e6536e85c0R396

I hope I'll be able to resume working on that JIRA soon, please watch the JIRA issue to be updated progress.

Explorer

Koji Kawamura, Did you get chance to work on the offset reset feature? Having ability to reset offset to a target value is necessary in case the pipeline needs to be reprocessed.

Hi @AViradia, No, I haven't. But I just found this Stackoverflow. If you're using ConsumeKafka, update the processor to use new consumer group id would be a simple work-around to reprocess from the beginning. http://stackoverflow.com/questions/37741936/how-to-delete-kafka-consumer-created-via-new-consumer-ap...

; ;