question Re: How do we achieve exactly-once processing in Nifi? in Archives of Support Questions (Read Only)

How do we achieve exactly-once processing in Nifi?

egarelnabi — Thu, 18 May 2017 22:37:56 GMT

I have a Nifi cluster consuming messages from Kafka and sending the output to a PostgresDB. How do I guarantee that each message will be consumed/processed exactly once?

Re: How do we achieve exactly-once processing in Nifi?

ssahi — Thu, 18 May 2017 22:50:00 GMT

Hi @Eyad Garelnabi

there is a great post about it here: https://community.hortonworks.com/questions/55454/can-nifi-promise-each-of-the-flowfiles-can-be-proc.html

Re: How do we achieve exactly-once processing in Nifi?

dchaffey — Thu, 18 May 2017 23:29:40 GMT

Here is a gross simplification that might be helpful: Exactly-once usually requires that the source system and destination system can somehow be made to agree on a method to manage at-least-once delivery and data-deduplication. NiFi could be the transport layer providing at-least-once delivery between the systems, but Kafka to NiFi without those semantics or some additional approach will not satisfy those requirements alone.