Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Need a mechanism to Sort flow files in a Queue.

avatar
Contributor

I have a large json which needs to be chunked into 2 MB bites using split record. 
Output flowflies from split record are ordered based on frament.index. 
There is a replaceText step right after SplitRecord where the order of flowfiles doesn't match with the fragment.index. 
I want the flowfiles in the queue to be ordered by frament.index
is there anyway we can achieve this?

1 REPLY 1

avatar

Hi,

Have you looked into the EnforceOrder processor :

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.23.2/org.apach...

Based on your description it should do the job. One thing to keep in mind if you want this to work is what is stating in the description :

"...

[IMPORTANT] In order to take effect of EnforceOrder, FirstInFirstOutPrioritizer should be used at EVERY downstream relationship UNTIL the order of FlowFiles physically get FIXED by operation such as MergeContent or being stored to the final destination.

"

The group identifier will be in this case the value of the fragment.identifier:

${fragment.identifie}

The Order Attribute will be the name of the fragment.index:

 fragment.index

If you find this helpful please accept the solution.

Thanks