Hi,
I am using NIFI to consume and publish data using kafka.
I have some queries on offset commit to ensure there is no data loss.
Currently as per NIFI logs, the consumerconfig shows that "enable.auto.commit= false", this means that the offset are committed manually.
Internal are these commits happening synchronously or asynchronously in NIFI? how to check this?
And also in offset we have two types in kafka
1. current offset (Sent records)- This is used to avoid resending same records again to the same consumer
2. committed offset (processed records) - It is used to avoid resending same records to a new consumer in the event of partition rebalance.
Regarding committed offset, This is kind of a indication sent back to kafka broker by consumer stating that the messages are consumed successfully? is my understanding right?
Any help would be appreciated.