The intent of this article is cover exactly what happens when a user clicks the "terminate" button on a processor component that has an actively running task.
Before we can discuss the "terminate" option, we need to understand a few basics about the NiFi application and a bit of history:
1. NiFi is a java application and the execution of any component (processors, controller service, reporting tasks, funnel, input/output ports, etc) happens within that single Java Virtual Machine (JVM) process. NiFi does not create a child process fro the execution of each component.
2. Since NiFi operates within a single JVM, it is not possible to "kill" a thread for an individual component without killing the entire JVM.
3. NiFi consists of well over 400 unique components and many of them are not executing native NiFi code. Many use client libraries not managed or controlled by NiFi. Others can be configured to execute command external to NiFi (ExecuteStreamCommand, ExecuteProcess, ExecuteScript, etc). Processors that invoke something external to NiFi's code base will result in a child process being created with its own pid. Keep in mind that processors of this type do not limit what externally is being invoked so take a generic approach to handling those child processes. The JVM invokes the external command and waits for it to respond complete.
4. Historically NiFi did not offer a terminate option since killing a thread in the NiFi JVM wis not possible. So when a component misbehaved (usually do to an issue external to NiFi code like network, client library hung, external command hung, etc), that NiFi component processor would get stuck just with the JVM thread waiting on that client library or external process to return. As such, the processor's concurrent task JVM thread is blocked. While you could select to stop the processor that would not help users get past the hung or long running thread. NiFi processors transition to a "stopping" state where it will remain until that library or task it is waiting on completes. Until that happens, users would not be able to modify the configuration or restart the component. This meant for truly hung issues the component would be blocked until the NiFi JVM was restarted.
5. As a result of the inconvenience/impact a hung thread causes, NiFi introduced the "terminate" option on a "stopped" component with an active thread.
What Actually happens when a user clicks "terminate":
1. "Terminate" is only possible when after a processor has been asked to stop and that stopped processor still has associated JVM thread running.
2. Since we know that killing a JVM thread is not possible without killing the entire JVM process (NiFi), the "terminate" option takes a different approach. When a processor executes, it is doing so typically in response to inbound queued FlowFile as the trigger. That means the inbound FlowFile is tied to the JVM thread that is executing. When the thread completes, that FlowFiles (or modified, cloned, new FlowFile depending on processor function) is moved to the appropriate outbound relationship of the processor.
3. So what the "terminate" function really does is releases the FlowFile associated to that running JVM thread back to the inbound connection, makes request to client library or external command to abort/exit, and then isolates that thread so that if it does actually complete post terminate, all returns are just sent to null.
4. When "terminate" has been selected, the UI will render the processors active threads differently to indicate if the processor has JVM threads that have been terminated but are still active.
NOTE: The number within the parenthesis denotes the current number of terminated threads still active.
5. If the client or external command responds to the request to exit, the active "terminated" thread will disappear. If not, it will continue to exist until thread finally completes or the entire NiFi JVM is restarted.
NOTE: A terminated thread has little impact on resources since a hung thread isn't consuming CPU. Now a long running CPU intensive thread may have impact.
6. Now that this "terminate" JVM thread has been isolated and any FlowFile(s) tied to that thread have been released to originating connection, users can modify the processor configuration and start the component processor again. When started again, the processor will execute again on the FlowFile(s) that once belonged to the terminated thread. So no dataloss is incurred as a result of using "terminate".
The "terminate" capability allows users to move on without needing to restart their NiFi JVM, thus reducing downtime and impact to other dataflows running on the NiFi canvas.
If you have a processor that constantly has hung process issues or has very long running threads, it is time to start looking at your source FlowFile(s), processor configuration, external command, or external service the processor may be waiting for a response from as possible sources of the issue.
Reference:
Apache NiFi Terminate documentation
