question Re: EnforceOrder Inactive Timeout not working in Support Questions

EnforceOrder Inactive Timeout not working

varungupta — Thu, 03 Oct 2024 13:15:05 GMT

Hello
I am using the EnforceOrder in order to sort the flow files. This procesosr is maintaning state and i want to clear in it some regular interval for e.g. 4 min. I am runnig NIFI over a cluster. I am clearing the state through Inactive Timeout propery by setting it 4 min. I can see some time state is clear and most of the time not. Please help in a resolution for how clear state consistently.

Re: EnforceOrder Inactive Timeout not working

MattWho — Thu, 03 Oct 2024 15:51:23 GMT

@varungupta

The EnforceOrder processor maintain state locally specific to each node in the cluster. When you right click on the processor an view state, you are seeing the reported retained local state form each node. The purge of state will only happen on a specific node where the criteria necessary for purge has been met.

Might be helpful to share your processor configuration here including properties and scheduling.

How are the Source FlowFiles being ingested and fed to this processor?

You can look at lineage on the EnforceOrder processor to see FlowFile history.

Thanks,

Matt

Re: EnforceOrder Inactive Timeout not working

varungupta — Thu, 03 Oct 2024 18:46:09 GMT

Hi @MattWho
Here is the configuration attached. I am setting Inactive Timeout as 2 min. Though after 10 min still the view state is not clear. Please refer to both screenshots. Flow is as below

ListFile-->UpdateAttribute for Order attribute-->EnforceOrder--> Further processing through other processor

If I send the same file again after Inactive Timeout then as the view state does not get clear it moves to the Skipped queue.

What I see as an observation is that if I send the flow file again after Inactive Timeout then the flowfile goes to the Skipped queue and at the same time the state gets clear. So, I need to do some special handling. please suggest

Re: EnforceOrder Inactive Timeout not working

MattWho — Fri, 04 Oct 2024 16:50:52 GMT

@varungupta

Tell me a bit more about your ListFile processor configuration.

Is the input directory a shared mount across all your nodes or does each node have a unique set of files in the input directory?
Is ListFile configured for execution of "All nodes" or "Primary node"?
How many files are being listed when ListFile is executing (Is it less then 10,000)?
How often is ListFile scheduled to run?
Is List File traversing sub-directories for files?

I assume you are extracting the sequenceNo from the filename.

As far as Enforce Order goes:

How are you handling the various relationships (routing via connection to where)?
Are you seeing FlowFiles routed to "overlook" relationship?

Matt