Member since
07-30-2019
3471
Posts
1642
Kudos Received
1020
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 162 | 06-03-2026 06:06 PM | |
| 460 | 05-06-2026 09:16 AM | |
| 830 | 05-04-2026 05:20 AM | |
| 499 | 05-01-2026 10:15 AM | |
| 626 | 03-23-2026 05:44 AM |
10-21-2022
12:28 PM
@Fredi A screenshot of the configuration of your UpdateAttribute processor including main configuration and configuration in the "Advanced" UI would be very helpful in understanding your setup and issue. Thanks, Matt
... View more
10-19-2022
08:20 AM
2 Kudos
The intent of this article is cover exactly what happens when a user clicks the "terminate" button on a processor component that has an actively running task. Before we can discuss the "terminate" option, we need to understand a few basics about the NiFi application and a bit of history: 1. NiFi is a java application and the execution of any component (processors, controller service, reporting tasks, funnel, input/output ports, etc) happens within that single Java Virtual Machine (JVM) process. NiFi does not create a child process fro the execution of each component. 2. Since NiFi operates within a single JVM, it is not possible to "kill" a thread for an individual component without killing the entire JVM. 3. NiFi consists of well over 400 unique components and many of them are not executing native NiFi code. Many use client libraries not managed or controlled by NiFi. Others can be configured to execute command external to NiFi (ExecuteStreamCommand, ExecuteProcess, ExecuteScript, etc). Processors that invoke something external to NiFi's code base will result in a child process being created with its own pid. Keep in mind that processors of this type do not limit what externally is being invoked so take a generic approach to handling those child processes. The JVM invokes the external command and waits for it to respond complete. 4. Historically NiFi did not offer a terminate option since killing a thread in the NiFi JVM wis not possible. So when a component misbehaved (usually do to an issue external to NiFi code like network, client library hung, external command hung, etc), that NiFi component processor would get stuck just with the JVM thread waiting on that client library or external process to return. As such, the processor's concurrent task JVM thread is blocked. While you could select to stop the processor that would not help users get past the hung or long running thread. NiFi processors transition to a "stopping" state where it will remain until that library or task it is waiting on completes. Until that happens, users would not be able to modify the configuration or restart the component. This meant for truly hung issues the component would be blocked until the NiFi JVM was restarted. 5. As a result of the inconvenience/impact a hung thread causes, NiFi introduced the "terminate" option on a "stopped" component with an active thread. What Actually happens when a user clicks "terminate": 1. "Terminate" is only possible when after a processor has been asked to stop and that stopped processor still has associated JVM thread running. 2. Since we know that killing a JVM thread is not possible without killing the entire JVM process (NiFi), the "terminate" option takes a different approach. When a processor executes, it is doing so typically in response to inbound queued FlowFile as the trigger. That means the inbound FlowFile is tied to the JVM thread that is executing. When the thread completes, that FlowFiles (or modified, cloned, new FlowFile depending on processor function) is moved to the appropriate outbound relationship of the processor. 3. So what the "terminate" function really does is releases the FlowFile associated to that running JVM thread back to the inbound connection, makes request to client library or external command to abort/exit, and then isolates that thread so that if it does actually complete post terminate, all returns are just sent to null. 4. When "terminate" has been selected, the UI will render the processors active threads differently to indicate if the processor has JVM threads that have been terminated but are still active. NOTE: The number within the parenthesis denotes the current number of terminated threads still active. 5. If the client or external command responds to the request to exit, the active "terminated" thread will disappear. If not, it will continue to exist until thread finally completes or the entire NiFi JVM is restarted. NOTE: A terminated thread has little impact on resources since a hung thread isn't consuming CPU. Now a long running CPU intensive thread may have impact. 6. Now that this "terminate" JVM thread has been isolated and any FlowFile(s) tied to that thread have been released to originating connection, users can modify the processor configuration and start the component processor again. When started again, the processor will execute again on the FlowFile(s) that once belonged to the terminated thread. So no dataloss is incurred as a result of using "terminate". The "terminate" capability allows users to move on without needing to restart their NiFi JVM, thus reducing downtime and impact to other dataflows running on the NiFi canvas. If you have a processor that constantly has hung process issues or has very long running threads, it is time to start looking at your source FlowFile(s), processor configuration, external command, or external service the processor may be waiting for a response from as possible sources of the issue. Reference: Apache NiFi Terminate documentation
... View more
Labels:
10-10-2022
07:20 AM
@knighttime Has your issue been resolved? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.
... View more
10-07-2022
07:51 AM
@MattWho Thank you very much, this worked perfectly for what I was needing. Can you tell me if this same flow also works with windows?
... View more
10-04-2022
12:13 AM
ResultInvoke in this case doesn't have any value, it's sent as a header with '${invokehttp.status.code}' as a value in http request. To assign status code to ResultInvoke use UpdateAttribute after the InvokeHTTP.
... View more
10-03-2022
01:59 AM
Hi and thanks for your reply. Integration with OIDC and NiFi it's no easy. I've tried you suggestion but don't works, seems the header don't follow the flow. But now, I've resolved using the certificate when I call the url. I've trusted zabbix certificate with nifi and use this curl: curl https://nificluster.info/nifi-api/flow/cluster/summary --insecure -H "Host: nificluster.info" --cert /pathcertificate/certificate_zabbix.pem --key /pathcertificate/certificate_zabbix.key Now I can check the cluster status "without" login.
... View more
09-30-2022
02:31 PM
@KD9 How long the NiFi server will validate a clients token is configured within the login-identity-providers.xml file via the following property: Authentication Expiration When setting up an automated process, using client tokens is not the best method. A better option would be to authenticate your client via a client certificate. With a client certificate, there is not need to obtain a token. That Client certificate will continue to work for the life of the certificate (certificates do have a valid until date set when you generate the certificate). So instead of passing a bearer token in your curl command, you would use your client pem key. The owner DN from the client certificate would be used as the user identity that you would then need to authorize in NiFi for the rest-api endpoint(s) needed for your automation. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
09-30-2022
02:19 PM
@John-MaxQ The MergeContent processor utilizes the Apache Commons Compress library which has a hard limit in tar size. There is an existing Apache NiFi jira for this here: https://issues.apache.org/jira/browse/NIFI-10273 If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
09-30-2022
02:04 PM
@ImranAhmed If I am understanding you correctly, you are trying to verify the content has correctly been modified before it is written out via your invokeHTTP processor.Inspecting that your dataflow is working as expected. If you stop your InvokeHTTP processor after the ReplaceText processor, the FlowFile processed by the ReplaceText will queue in the connection. You can then right click on the connection and list the queue. From there you can view/download the content of that FlowFile post ReplaceText to verify it contains the expected modified content before the InvokeHTTP is started and writes it to your DB. Thanks, Matt
... View more
09-29-2022
08:48 PM
Hello everyone, the problem was solved by replacing java Oracle openjdk-11.0.1.13 with java Liberica jdk11.0.16
... View more