Created 05-30-2024 08:08 AM
Hello Experts,
I see this red color high lighted number "2(1)"on Apache nifi processor.
Is this something related to background process (processor thread) failing or something?
I face the issue of this custom processor getting stuck once in a while, trying to understand the issue.
This processor just invokes an http post endpoint to upload a file.
Any help/suggestion is appreciated.
Thanks,
Mahendra
Created on 05-30-2024 10:00 AM - edited 05-30-2024 10:02 AM
@hegdemahendra
The small number in upper right corner of any processor shows the number of active threads at time the UI was last refreshed. The default auto refresh of the UI is every 30 seconds.
It turns red when their is an active terminated thread. So with your example above 2(1), it is telling you that this processor as 2 active threads and 1 terminated thread.
A terminated thread is the result of manual user intervention. When a processor asked to change run-status from "running" to "stopped", (Stopping Component) it first transition into a state of "stopping". It does not transition to "stopped" until all active threads complete. NiFi provides and option to "terminate" when in a stopping state because of active threads.
Terminate (Terminating a components tasks) does not kill that active thread since all thread belong to a single JVM. What the terminate function does is release any FlowFile tied to the active thread(s) back to their originating connection and marks the thread as terminated. That terminated thread will continue to execute until it completes or the JVM is restarted. Should that now "terminated" thread complete, all output is sent to dev null instead of resulting in any down stream movement.
This allows users to handle scenarios where there are long running threads or hung threads preventing the stopping, changing of configuration, and starting of a processor.
When a terminated processor is restarted it will re-process the FlowFile(s) that were originally tied to the terminated thread(s). This prevents any data loss from occurring.
If a terminated thread is in a permanently hung state, the only way to get rid of it completely is a restart of NiFi which will kill the JVM after a graceful shutdown period.
As far as your custom processor getting stuck, you would need to collect thread dumps and inspect those to see what your thread is waiting on that is blocking it from progressing and address that in your custom code.
Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.
Thank you,
Matt
Created on 05-30-2024 10:00 AM - edited 05-30-2024 10:02 AM
@hegdemahendra
The small number in upper right corner of any processor shows the number of active threads at time the UI was last refreshed. The default auto refresh of the UI is every 30 seconds.
It turns red when their is an active terminated thread. So with your example above 2(1), it is telling you that this processor as 2 active threads and 1 terminated thread.
A terminated thread is the result of manual user intervention. When a processor asked to change run-status from "running" to "stopped", (Stopping Component) it first transition into a state of "stopping". It does not transition to "stopped" until all active threads complete. NiFi provides and option to "terminate" when in a stopping state because of active threads.
Terminate (Terminating a components tasks) does not kill that active thread since all thread belong to a single JVM. What the terminate function does is release any FlowFile tied to the active thread(s) back to their originating connection and marks the thread as terminated. That terminated thread will continue to execute until it completes or the JVM is restarted. Should that now "terminated" thread complete, all output is sent to dev null instead of resulting in any down stream movement.
This allows users to handle scenarios where there are long running threads or hung threads preventing the stopping, changing of configuration, and starting of a processor.
When a terminated processor is restarted it will re-process the FlowFile(s) that were originally tied to the terminated thread(s). This prevents any data loss from occurring.
If a terminated thread is in a permanently hung state, the only way to get rid of it completely is a restart of NiFi which will kill the JVM after a graceful shutdown period.
As far as your custom processor getting stuck, you would need to collect thread dumps and inspect those to see what your thread is waiting on that is blocking it from progressing and address that in your custom code.
Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.
Thank you,
Matt
Created 05-30-2024 11:24 PM
What an explanation ! Cleared my doubts.
Thank you so much @MattWho .