Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Ideas for In Order Processing in NiFi

Solved Go to solution

Ideas for In Order Processing in NiFi

Explorer

I have a few nifi boxes and they are load balanced using HAproxy. In our architecture we decided not to use NiFi Clustering.

I have some data feeds that require In Order Delivery of files (xml). Since their data is load balanced across multiple servers, I cannot guarantee that the files will be delivered in the order that they were received.

Any Ideas for how I could achieve this? Is there an attribute I could create? and a processor that maintains order? and waits for files of certain values to be in a queue before delivering them? Maybe a groovy script that could accomplish this (even an example of a groovy script that pays attention to attributes and multiple files would be helpful?

John

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Ideas for In Order Processing in NiFi

Guru

@John T

apart from using FIFO priorotizization config on all of your connections, have you looked at the EnforceOrder processor in the latest version of NiFi? I think it does what you want? https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.3.0/org.apache...

View solution in original post

3 REPLIES 3
Highlighted

Re: Ideas for In Order Processing in NiFi

Guru

@John T

apart from using FIFO priorotizization config on all of your connections, have you looked at the EnforceOrder processor in the latest version of NiFi? I think it does what you want? https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.3.0/org.apache...

View solution in original post

Highlighted

Re: Ideas for In Order Processing in NiFi

Explorer

That's exactly what we needed! Thank you!!!

Highlighted

Re: Ideas for In Order Processing in NiFi

Contributor

Even I am exploring ways to deliver data in order via NIFi. Please note that even on single node your data can get out of order if NiFi penalizes a fiow file or if you try to implement some custom "retry" logic.

Don't have an account?
Coming from Hortonworks? Activate your account here