question Re: Ideas for In Order Processing in NiFi in Archives of Support Questions (Read Only)

Ideas for In Order Processing in NiFi

johnmteabo — Tue, 05 Sep 2017 10:04:15 GMT

I have a few nifi boxes and they are load balanced using HAproxy. In our architecture we decided not to use NiFi Clustering.

I have some data feeds that require In Order Delivery of files (xml). Since their data is load balanced across multiple servers, I cannot guarantee that the files will be delivered in the order that they were received.

Any Ideas for how I could achieve this? Is there an attribute I could create? and a processor that maintains order? and waits for files of certain values to be in a queue before delivering them? Maybe a groovy script that could accomplish this (even an example of a groovy script that pays attention to attributes and multiple files would be helpful?

John

Re: Ideas for In Order Processing in NiFi

ssahi — Tue, 05 Sep 2017 22:18:32 GMT

@John T

apart from using FIFO priorotizization config on all of your connections, have you looked at the EnforceOrder processor in the latest version of NiFi? I think it does what you want? https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.3.0/org.apache.nifi.processors.standard.EnforceOrder/index.html

Re: Ideas for In Order Processing in NiFi

mayank_rathi — Tue, 05 Sep 2017 23:14:08 GMT

Even I am exploring ways to deliver data in order via NIFi. Please note that even on single node your data can get out of order if NiFi penalizes a fiow file or if you try to implement some custom "retry" logic.

Re: Ideas for In Order Processing in NiFi

johnmteabo — Sat, 09 Sep 2017 02:22:57 GMT

That's exactly what we needed! Thank you!!!