Flowfile stuck in Wait in EnforceOrder processor

[#1]

Hi, I am trying to set up a workflow in Nifi, where I need to process Flowfiles in a specific order. To my understand, this is what the EnforceOrder processor was designed for.

The part I am having problems with looks like this:

In Set Merge Priority I am setting a Merge Priority attribute (1 for the left node, 2 for the right node). I am also setting an Enforce property to group the EnforceOrder by:

My EnforceOrder processor is configured as followed:

Only flowfiles with merge_priority = 1 will be successfully routed, while the flowfiles with merge_priority = 2 are stuck in "Wait". Files with merge_priority = 2 to arrive at the EnforceOrder processor before the flowfiles with merge_priority = 1. To my understanding, the flowfiles should be re-processed as soon as a file with the current merge_priority has been processed?

[#2 · Accepted solution]

@Chram 

Lets backup and first understand the use case you are trying to solve here.

From the look of your dataflows it appears you have two FlowFiles that you want to merge via the MergeContent processor.  The order in which they are merged is important to you.  This sounds like a use case better solved by using the "Defragment" merge strategy in the MergeContent processor rather then trying to use EnforceOrder processor.

Is there more to your use case which necessitates the use of enforceOrder?

As far as your dataflow goes,  I am having trouble reproducing the issue you described.
I see my priority 2 file sitting in wait only until my matching priority 1 file arrives. 
The dataflow screenshot you shared also does not show a wait relationship being routed anywhere.  Are you using the "retry" + "terminate" checkboxes on the wait relationship instead?  If so, what are the retry settings configured?

Rather then use "retry" on the wait relationship, try dragging a new connection away from and back to the EnforceOrder processor to create a loop and assign the "Wait" relationship to that looped connection (make sure to also uncheck "retry" on the wait relationship).   Does same issue still persist?

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#3]

Hi Matt,
thanks for your your answer.

---

@MattWho wrote:

From the look of your dataflows it appears you have two FlowFiles that you want to merge via the MergeContent processor.  The order in which they are merged is important to you.  This sounds like a use case better solved by using the "Defragment" merge strategy in the MergeContent processor rather then trying to use EnforceOrder processor.

---

Sorry for not mentioning this. My workflow starts with a series of API calls to obtain data, that is converted into xml fragments. With the merge processor, I am trying to merge those xml fragments into one final XML file. Hence the order is important.
The files can have different product_id attributes and I only want to merge files with the same product id. Therefore I am using the MergeProcessor with the Bin-Packing Algorithm. As far as I know (I just started using Nifi last month) it is not possible to control the order using this algorithm?

---

@MattWho wrote:

As far as your dataflow goes,  I am having trouble reproducing the issue you described.
I see my priority 2 file sitting in wait only until my matching priority 1 file arrives. 
The dataflow screenshot you shared also does not show a wait relationship being routed anywhere.  Are you using the "retry" + "terminate" checkboxes on the wait relationship instead?  If so, what are the retry settings configured?

---

I hadn't configured the wait relationship yet. I was under the impression, the EnforceOrder processor would re-visit flowfiles in "wait" status once other priority 1 files have been processed.

I routed the wait relationship to a RouteOnAttribute processor (is there another way?) which then routes back to EnforceOrder:

The priority 2 file will now be processed, but the order, in which the files are merged, seems to be random. Sometimes its the correct order, sometimes its not.

[#4]

I changed my config to this:

This seems to do the trick.