Skip to main content
Shrink
Shrink
Silver ⭐️⭐️⭐️
February 24, 2025
Question

Nifi why different set of records from ScriptedFilterRecord ?

Hello, 
i am using ScriptedFilterRecord to filter empty records for one of the record filed. It is working as expected but when i try to push same records to QueryRecord to get list down those ids. NIFi push different set of records to query record processor. Please refer attached screenshot. 

Shrink_0-1740427812718.png

Thank you.

3 replies

MattWho
Titanium ⭐️⭐️⭐️
MattWho
Titanium ⭐️⭐️⭐️
February 25, 2025

@Shrink 

Welcome to the community.

Unfortunately,  your image attachment is not present for review and there is not enough detail to provide any detailed suggestions here.

NiFi processors like ScriptedFilterRecord and QueryRecord execute against the content of the NiFi FlowFile from the upstream connection.  The ScriptedFilterRecord processor has three possible downstream relationships (Success, Failure, and Original). 

  1. I guess my first question would be how are you routing these three relationships?
  2. Assuming your downstream connection does not contain both the original and success relationships in it, have you inspected the content on the connection FlowFiles prior to starting the QueryRecord processor to make sure it is what you expect?

Please help our community grow and thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate
MattWho
Titanium ⭐️⭐️⭐️
MattWho
Titanium ⭐️⭐️⭐️
February 26, 2025

@Shrink 

I see your image is now showing in the original query.

I see two connection "success" and one connection "Failure".  I assume you are auto-terminating the "original" relationship on the ScriptedFilterRecord processor?

I am confused by you would loop the "success" relationship in a connection back to the ScriptedFilterRecord processor.  All this will do is create an endless loop of reprocessing the same "success" FlowFiles over and over again indefinitely.

I see your two arrows pointing at the two queues with different counts, but based on your design with the looped "success" relationship. this would be expected.  For every FlowFile ingested from the upstream "Original" connection, a new child FlowFile is created and sent to the "success" relationships.  One of the two FlowFiles produced will be a clone of the other.  With one copy going to each "success" connection.  The looped "success" connection becomes another inbound connection to the ScriptedFilterRecord processor. So those already filtered FlowFiles are processed and once again a child and clone of that child is created.  So you are exponentially replicating your FlowFiles as they are being sent to the QueryRecord processor, thus the much higher FlowFile count on that downstream connection.

Eliminating the looped "success" connection (assuming it contains the success relationship) would stop the data cloning and constant reprocessing of the same content over and over again.

 

Please help our community grow and thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate
Shrink
ShrinkAuthor
Silver ⭐️⭐️⭐️
February 26, 2025

Thank you @MattWho . 
Thanks for brief explanation. I understand the loop is causing large number of queue. Let me redesign the flow. 

Thanks ! 

Translate
Terms & ConditionsAccessibility statement