Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Ideas for In Order Processing in NiFi

avatar
Rising Star

I have a few nifi boxes and they are load balanced using HAproxy. In our architecture we decided not to use NiFi Clustering.

I have some data feeds that require In Order Delivery of files (xml). Since their data is load balanced across multiple servers, I cannot guarantee that the files will be delivered in the order that they were received.

Any Ideas for how I could achieve this? Is there an attribute I could create? and a processor that maintains order? and waits for files of certain values to be in a queue before delivering them? Maybe a groovy script that could accomplish this (even an example of a groovy script that pays attention to attributes and multiple files would be helpful?

John

1 ACCEPTED SOLUTION

avatar
Guru

@John T

apart from using FIFO priorotizization config on all of your connections, have you looked at the EnforceOrder processor in the latest version of NiFi? I think it does what you want? https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.3.0/org.apache...

View solution in original post

3 REPLIES 3

avatar
Guru

@John T

apart from using FIFO priorotizization config on all of your connections, have you looked at the EnforceOrder processor in the latest version of NiFi? I think it does what you want? https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.3.0/org.apache...

avatar
Rising Star

That's exactly what we needed! Thank you!!!

avatar
Contributor

Even I am exploring ways to deliver data in order via NIFi. Please note that even on single node your data can get out of order if NiFi penalizes a fiow file or if you try to implement some custom "retry" logic.