Skip to main content
varungupta
varungupta
Silver ⭐️⭐️
October 3, 2024
Question

EnforceOrder  Inactive Timeout not working

Hello 
I am using the EnforceOrder in order to sort the flow files. This procesosr is maintaning state and i want to clear in it some regular interval for e.g. 4 min. I am runnig NIFI over a cluster. I am clearing the state through Inactive Timeout propery by setting it 4 min. I can see some time state is clear and most of the time not. Please help in a resolution for how clear state consistently.

3 replies

MattWho
Titanium ⭐️⭐️⭐️
MattWho
Titanium ⭐️⭐️⭐️
October 3, 2024

@varungupta 

The EnforceOrder processor maintain state locally specific to each node in the cluster.  When you right click on the processor an view state, you are seeing the reported retained local state form each node.  The purge of state will only happen on a specific node where the criteria necessary for purge has been met.

Might be helpful to share your processor configuration here including properties and scheduling.

How are the Source FlowFiles being ingested and fed to this processor?

You can look at lineage on the EnforceOrder processor to see FlowFile history.

Thanks,

Matt

Translate
varungupta
varunguptaAuthor
Silver ⭐️⭐️
October 3, 2024

Hi @MattWho 
Here is the configuration attached. I am setting Inactive Timeout as 2 min. Though after 10 min still the view state is not clear. Please refer to both screenshots. Flow is as below 

ListFile-->UpdateAttribute for Order attribute-->EnforceOrder--> Further processing through other processor

If I send the same file again after Inactive Timeout then as the view state does not get clear it moves to the Skipped queue.
What I see as an observation is that if I send the flow file again after Inactive Timeout then the flowfile goes to the Skipped queue and at the same time the state gets clear. So, I need to do some special handling. please suggest

EnforceOrder_Inactive_Timeout.pngEnforceOrderStatenotclearing.png

Translate
MattWho
Titanium ⭐️⭐️⭐️
MattWho
Titanium ⭐️⭐️⭐️
October 4, 2024

@varungupta 

Tell me a bit more about your ListFile processor configuration.  

  1. Is the input directory a shared mount across all your nodes or does each node have a unique set of files in the input directory?
  2. Is ListFile configured for execution of "All nodes" or "Primary node"?
  3. How many files are being listed when ListFile is executing (Is it less then 10,000)?
  4. How often is ListFile scheduled to run?
  5. Is List File traversing sub-directories for files?

I assume you are extracting the sequenceNo from the filename.

As far as Enforce Order goes:

  1. How are you handling the various relationships (routing via connection to where)?
  2. Are you seeing FlowFiles routed to "overlook" relationship?

Matt

Translate
    Terms & ConditionsAccessibility statement