Nifi ConsumeKafka processor

[#1]

Hi,

Could someone shed some light on how the above processor initiates the commit? Is there a way to enforce commit only after a set of subsequent processors completes successfully, for e.g. PutHDFS?

[#2 · Accepted solution]

ConsumeKafka commits the offsets to Kafka right after the data has been written to flow file and the session for that flow flow has been committed. This way there is no chance for the data to be lost before committing the offsets to Kafka because the data has already been persisted to NiFi's repositories.

Currently there is not a concept of having a series of processors treated as one operation. Right now you can think of it as two separate transfers of data, the first being from Kafka to NiFi, the second from NiFi to HDFS.