NiFi-Kafka: Commit failure notification due to rebalance

[#1]

I only see success relationship available for ConsumeKafka processor. In case of a failure, that is processor failed to commit the batch due to rebalance do we have any workaround/option to redirect to a PutEmail processor?

[#2]

If it fails to commit the batch, then the data is still in Kafka because the offset wasn't updated, so it would be pulled again next time after the rebalance.

[#3]

Yes, but if we need to track how many times the consumer has been going in retries, and notify whenever that happens? Since I do not see a failure relationship for the ConsumeKafka processor, do we have any other workaround/options to achieve this?

This is to ensure that we prevent duplicates by identifying the cause and notify whenever that occurs.

[#4 · Accepted solution]

ConsumeKafka is implemented with "at-least once" guarantees, so it writes the data to a flow file and commits the session for that flow file, before committing the offsets to ensure no data loss. If an error happens committing the offsets, as can happen with the rebalance, the flow file is already committed and transferred to success and can't be undone.

If it was done in the reverse order (commit offsets first, then commit the flow file session) you could route to a failure if committing the offsets failed, but you would also risk data loss if NiFi crashed after commit offsets but before committing the flow file session...this would be "at-most once" guarantees.