Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How do we achieve exactly-once processing in Nifi?

Solved Go to solution
Highlighted

How do we achieve exactly-once processing in Nifi?

I have a Nifi cluster consuming messages from Kafka and sending the output to a PostgresDB. How do I guarantee that each message will be consumed/processed exactly once?

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: How do we achieve exactly-once processing in Nifi?

Guru
2 REPLIES 2
Highlighted

Re: How do we achieve exactly-once processing in Nifi?

Guru

Re: How do we achieve exactly-once processing in Nifi?

Here is a gross simplification that might be helpful: Exactly-once usually requires that the source system and destination system can somehow be made to agree on a method to manage at-least-once delivery and data-deduplication. NiFi could be the transport layer providing at-least-once delivery between the systems, but Kafka to NiFi without those semantics or some additional approach will not satisfy those requirements alone.

Don't have an account?
Coming from Hortonworks? Activate your account here