Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Query on kafka offset commit in NIFI

avatar
Explorer

Hi,

I am using NIFI to consume and publish data using kafka.

 

I have some queries on offset commit to ensure there is no data loss.

 

Currently as per NIFI logs, the consumerconfig shows that "enable.auto.commit= false", this means that the offset are committed manually.

Internal are these commits happening synchronously or asynchronously in NIFI? how to check this?

 

And also in offset we have two types in kafka

 

1. current offset (Sent records)- This is used to avoid resending same records again to the same          consumer

2. committed offset (processed records) - It is used to avoid resending same records to a new consumer in the event of partition rebalance.

 

Regarding committed offset, This is kind of a indication sent back to kafka broker by consumer stating  that the messages are consumed successfully? is my understanding right?

 

Any help would be appreciated.

 

0 REPLIES 0