Support Questions
Find answers, ask questions, and share your expertise

Ideas for In Order Processing in NiFi

Solved Go to solution
Highlighted

Ideas for In Order Processing in NiFi

Explorer

I have a few nifi boxes and they are load balanced using HAproxy. In our architecture we decided not to use NiFi Clustering.

I have some data feeds that require In Order Delivery of files (xml). Since their data is load balanced across multiple servers, I cannot guarantee that the files will be delivered in the order that they were received.

Any Ideas for how I could achieve this? Is there an attribute I could create? and a processor that maintains order? and waits for files of certain values to be in a queue before delivering them? Maybe a groovy script that could accomplish this (even an example of a groovy script that pays attention to attributes and multiple files would be helpful?

John

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Ideas for In Order Processing in NiFi

Guru

@John T

apart from using FIFO priorotizization config on all of your connections, have you looked at the EnforceOrder processor in the latest version of NiFi? I think it does what you want? https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.3.0/org.apache...

View solution in original post

3 REPLIES 3
Highlighted

Re: Ideas for In Order Processing in NiFi

Guru

@John T

apart from using FIFO priorotizization config on all of your connections, have you looked at the EnforceOrder processor in the latest version of NiFi? I think it does what you want? https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.3.0/org.apache...

View solution in original post

Re: Ideas for In Order Processing in NiFi

Explorer

That's exactly what we needed! Thank you!!!

Highlighted

Re: Ideas for In Order Processing in NiFi

Contributor

Even I am exploring ways to deliver data in order via NIFi. Please note that even on single node your data can get out of order if NiFi penalizes a fiow file or if you try to implement some custom "retry" logic.

Don't have an account?