How do we achieve exactly-once processing in Nifi?

[#1]

I have a Nifi cluster consuming messages from Kafka and sending the output to a PostgresDB. How do I guarantee that each message will be consumed/processed exactly once?

[#2 · Accepted solution]

Hi @Eyad Garelnabi

there is a great post about it here: https://community.hortonworks.com/questions/55454/can-nifi-promise-each-of-the-flowfiles-can-be-proc...

[#3]

Here is a gross simplification that might be helpful: Exactly-once usually requires that the source system and destination system can somehow be made to agree on a method to manage at-least-once delivery and data-deduplication. NiFi could be the transport layer providing at-least-once delivery between the systems, but Kafka to NiFi without those semantics or some additional approach will not satisfy those requirements alone.