https://community.cloudera.com/t5/Archives-of-Support-Questions/Can-nifi-promise-each-of-the-flowfiles-can-be-processed/m-p/141657#M40069 <P>Thanks very much!</P> Fri, 09 Sep 2016 13:16:56 GMT zhangweigang20 2016-09-09T13:16:56Z https://community.cloudera.com/t5/Archives-of-Support-Questions/Can-nifi-promise-each-of-the-flowfiles-can-be-processed/m-p/141655#M40067 <P>Can nifi promise each of the flowfiles can be processed correctly, no missing and no duplicate ? </P><P>How the nifi realize it.</P><P>Thanks for your reply.</P> Thu, 08 Sep 2016 07:43:55 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Can-nifi-promise-each-of-the-flowfiles-can-be-processed/m-p/141655#M40067 zhangweigang20 2016-09-08T07:43:55Z https://community.cloudera.com/t5/Archives-of-Support-Questions/Can-nifi-promise-each-of-the-flowfiles-can-be-processed/m-p/141656#M40068 <P><A rel="user" href="https://community.cloudera.com/users/11862/zhangweigang20.html" nodeid="11862">@David DN</A> what you are describing here is notion of "Exactly Once Delivery." I would refer you to <A href="http://bravenewgeek.com/you-cannot-have-exactly-once-delivery/" target="_blank">http://bravenewgeek.com/you-cannot-have-exactly-once-delivery/</A> in order to get an understanding of why this is actually not possible in any distributed system.</P><P>Often what we hear people discussing is the notion of "Exactly Once semantics" in order to overcome this. However, the notion of Exactly Once semantics can be achieved between two systems only if the sending system can guarantee At Least Once delivery and the receiving side provides a mechanism for data de-duplicaiton.</P><P>When NiFi receives data from an external source, it does provide the capability for data de-duplication via the DetectDuplicate processor. So you can construct your flow so that if you receive data multiple times, you will process it only once. However, this is only achieved if you are receiving data over a reliable channel (for instance, ListenUDP may drop data as the UDP protocol is inherently lossy).</P><P>NiFi generally will guarantee At Least Once delivery of your data (I say generally because it depends on the processor. For instance, the PutKafka processor will provide At Least Once delivery if configured to do so but if configured as Best Effort delivery, it may not) wen sending to an external system. However, to ensure that data is not duplicated on the receiving system, it would require that the receiving system also have some way to de-duplicate data.</P> Thu, 08 Sep 2016 08:36:57 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Can-nifi-promise-each-of-the-flowfiles-can-be-processed/m-p/141656#M40068 mpayne 2016-09-08T08:36:57Z https://community.cloudera.com/t5/Archives-of-Support-Questions/Can-nifi-promise-each-of-the-flowfiles-can-be-processed/m-p/141657#M40069 <P>Thanks very much!</P> Fri, 09 Sep 2016 13:16:56 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Can-nifi-promise-each-of-the-flowfiles-can-be-processed/m-p/141657#M40069 zhangweigang20 2016-09-09T13:16:56Z