Ideas for In Order Processing in NiFi

[#1]

I have a few nifi boxes and they are load balanced using HAproxy. In our architecture we decided not to use NiFi Clustering.

I have some data feeds that require In Order Delivery of files (xml). Since their data is load balanced across multiple servers, I cannot guarantee that the files will be delivered in the order that they were received.

Any Ideas for how I could achieve this? Is there an attribute I could create? and a processor that maintains order? and waits for files of certain values to be in a queue before delivering them? Maybe a groovy script that could accomplish this (even an example of a groovy script that pays attention to attributes and multiple files would be helpful?

John

[#2 · Accepted solution]

@John T

apart from using FIFO priorotizization config on all of your connections, have you looked at the EnforceOrder processor in the latest version of NiFi? I think it does what you want? https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.3.0/org.apache...

[#3]

That's exactly what we needed! Thank you!!!

[#4]

Even I am exploring ways to deliver data in order via NIFi. Please note that even on single node your data can get out of order if NiFi penalizes a fiow file or if you try to implement some custom "retry" logic.