Member since
07-30-2019
3473
Posts
1642
Kudos Received
1021
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 308 | 06-15-2026 08:08 AM | |
| 515 | 06-03-2026 06:06 PM | |
| 752 | 05-06-2026 09:16 AM | |
| 1748 | 05-04-2026 05:20 AM | |
| 840 | 05-01-2026 10:15 AM |
10-24-2022
08:59 AM
@MrBurns You want to take the URL that written to the FlowFile's attribute "http.request.uri" and generate a Json, correct? Where do you want to write that JSON (a new FlowFile attribute? Content of the FlowFile?)? There are multiple ways to handle this. If you just want to write JSON to a new FlowFile Attribute, you could use the "Advanced" UI of the UpdateAttribute by setting up a rule for each url type. If you want to write to the content of a FlowFile, you could follow the above UpdateAttribute with a replaceText processor that does an "always replace" to write the json from the attribute to the content of the FlowFile. another option here is to use a RouteOnAttribute to route each url type to a unique ReplaceText to handle the specific url type. I like first option since you can easily add new rules to the UpdateAttribute if any additional URL types are introduced without needing to modify the rest of your dataflow. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
10-24-2022
07:35 AM
@D5ha Your issue here is with the certificate being used to perform the clientAuth action. Your certificate would also not work if you had a multi-node cluster. It is only working as single node cluster because there are no other nodes for which your single node need to communicate as a client. The keystore requirements for NiFi are as follows: 1. keystore MUST contain ONLY one PrivateKeyEntry 2. PrivateKeyEntry MUST have both clientAuth and ServerAuth ExtendedKeyUsage (EKU) 3. PrivateKeyEntry MUST have a SubjectAlternativeName (SAN) entry that matches the NiFi node's server hostname. If you are also going to be addressing your server by its IP, you should have that IP as a SAN entry as well. Any other alternative hostname this server may be known as (meaning user type that alternate hostname in a URL to reach this host) should also be added to SAN. In your case, the current issue happens in the mutual TLS handshake. You have configured your SiteToSiteBulletinReportingTask to send to https://<some ip>/nifi. The same NiFi server receive that client hello and responds with a server hello back which includes the SAN entries. In your case the client (reporting task) looks at that server hello and basically rejects the handshake at that point in time. It does this because of what looks like a man-in-the-middle attack. The client tried to reach host <some ip> but instead a host with san <localhost> responded. There is no configuration change you can make in your secured NiFi to get around this. You'll need to get a new certificate meeting that above min criteria I outlined. You'll need t do this also if you ever intend to add more hosts to your NiFi cluster. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
10-21-2022
01:25 PM
@DGaboleiro I am a bit confused by yoru dataflow design. In a NiFi multi-node cluster, each node is only aware of and can only execute upon FlowFiles present on that one node. So in your Dataflow you have the QueryCasandra processor executing on "primary node" only as you should (having it execute on all nodes would result in both your nodes performing same query and returning same data). You then Split that Json and use a DistributeLoad processor for what appears to me as means to then send some FlowFIle to node 1 and other half to node 2. This is not the best way to do this. You are running Apache NiFi 1.17 which means that load balanced connections are possible that can accomplish the same without all these additional processors. https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#settings After your FlowFiles (this is what is being moved from processor to processor on your canvas) have been distributed I see that you are a MergeContent processor. The MergeContent processor can only merge the FlowFiles present on the same node. It will not merge FlowFiles from multiple nodes to a single FlowFile. So if your desire is to have one merge of all FlowFiles, distributing them across multiple nodes will not give you that desired outcome. You should never configure any processor that accepts an inbound connection for "primary node" only execution. This is important since which node is elected as primary node can change at anytime. Execution strategy has nothing to do with the availability of FlowFiles on each node on which to execute. What is important to understand is that each node in yoru NiFi cluster has its own copy of the Flow, its own set of Content and FlowFile repositories contain unique data, and each nodes executes the processors in its flow with no regard of the existence of other nodes. A node is simply aware from Zookeeper if it has been elected as the cluster coordinator and/or primary node. If it is elected primary node, it will execute "primary node" and "all nodes" components. If it is not the primary node, it will only execute the "all nodes" components. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
10-21-2022
12:56 PM
@rangareddyy What is important to understand is that the NiFi component processors are not being executed by the user authenticated (assuming secured NiFi) into NiFi, but rather by the NiFi service user. So let's say that your NiFi service is owned by a "nifiservice" linux account. Whatever umask is configured for that user will be applied to directories and files create by that user. Now if your script is using sudo, it is changing the user that executes your script resulting in different user ownership and permission from the "nifiservice" user. Subsequent component processors will also execute as the "nifiservice" user and then not have access to those files and directories. So you'll need to take this in to account as you built your scripts. Make sure that your scripts are adjusting permissions on the directory tree and files as needed so your "nifiservice" user or all users can access the files needed downstream in your dataflows. So in yoru case it sounds like your script executed by ExecuteScript processor is creating a sh file not owned by the "nifiservice" user or does not have execute permission set on it. The ExecuteStreamCommand processor will attempt to execute the sh command on disk as the "nifiservice" user only. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
10-21-2022
12:41 PM
1 Kudo
@Jagapriyan As a daily job, i may suggest you tackle this differently. You know your source files are written between 8am - 9am each day. So i would configure your listSFTP to run on a cron schedule so it runs every second from 9am-10am to make sure all files are listed. Then knowing that your files may number 90+ (unknown on max) , I would configure your "Min Num of Entries" to some value you know the count will never reach. Make sure "Max Num Entries" is set to a value higher than that. Then configure the "Max Bin Age" to some time 30 mins? What this does is allow MergeContent to continue to allocated FlowFiles to a bin for 30 minutes at which time the bin is forced to merge even if the min value has not be reached. Doing this makes sure you get only one FlowFile out per bin per node. That single FlowFile can then be used to trigger your putEmail used for notification. Additionally, the merged FlowFile will have an attribute "merge.count" added that you can use in your email body to report number of FlowFiles that were ingested. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
10-21-2022
12:28 PM
@Fredi A screenshot of the configuration of your UpdateAttribute processor including main configuration and configuration in the "Advanced" UI would be very helpful in understanding your setup and issue. Thanks, Matt
... View more
10-21-2022
12:23 PM
1 Kudo
@DGaboleiro That is not me as the assignee to jira https://issues.apache.org/jira/browse/NIFI-8043. But that Matt is an awesome guy @mburgess. Thanks, Matt
... View more
10-21-2022
12:21 PM
@RRosa I am not clear on what you mean by "migrating the flow files"? A NiFi FlowFile is the object that is traversed via connections between NiFi component processors on the NiFi canvas. Are you talking about migrating your actively queued FlowFiles from NiFi cluster 1 (Apache NiFi 1.12.1) to NiFi cluster 2 (Apache NiFi 1.17.0)? Or are you talking about migrating the flow.xml.gz file (contains everything you have configured on the canvas of yoru NiFi) from old cluster to new? General guidance for upgrading Apache NiFi can be found in the admin guide here: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#upgrading_nifi The only thing I see NOT covered in that guidance is the preservation of component state. Within a cluster Component state may be stored depending on component in either a local state directory on each node (each node holds only state for that node) or stored in cluster state (written to ZK and shared across all nodes). Now if you are installing the new version of NiFi on the same hosts where the old NiFi nodes were running, simply preserve the state configuration and the new nodes when started with a copy of the flow.xml.gz will continue to read and use same state. Same goes for new nodes using the same external ZK that the previous nodes used (stop old before starting new hosts). While the Documentation recommends that you process out all queued FlowFiles from cluster 1 before starting cluster 2 that is not required. If new nodes point to same content, flowfile, and provenance repositories as previous node that data will get loaded back in on startup and processing continue where it left off. Remember that each nodes repositories are unique to that node (meaning you can't combine them and they don't all contain the same content). Another thing to review is the release notes. https://cwiki.apache.org/confluence/display/NIFI/Release+Notes You'll want to review all the release notes between 1.12 and 1.17. Apache NiFi is known to deprecate and remove some components (processors, controller services, reporting tasks, etc) from time to time. You'll want to check to see if any components you use in your current dataflows are being removed. Additional some components may have changed typically resulting in additional properties being added. When you start with the newer version of NiFi, it will load your existing flow.xml.gz (1.17 will actually generate a flow.json.gz file from your flow.xml.gz) and upgrade all your components to use the newer 1.17 version of the component classes. So you'll want to review you flow after upgrade to make sure none of your components that were previous valid have become invalid because new property exist that must be configured. NOTE: 1.17 will use start using the flow.json.gz once upgrade, as the flow.xml.gz format is deprecated. If you found this response assisted you with your query, please take a moment to login and click on "Accept as Solution" below this response. Thank you, Matt
... View more
10-19-2022
08:20 AM
2 Kudos
The intent of this article is cover exactly what happens when a user clicks the "terminate" button on a processor component that has an actively running task. Before we can discuss the "terminate" option, we need to understand a few basics about the NiFi application and a bit of history: 1. NiFi is a java application and the execution of any component (processors, controller service, reporting tasks, funnel, input/output ports, etc) happens within that single Java Virtual Machine (JVM) process. NiFi does not create a child process fro the execution of each component. 2. Since NiFi operates within a single JVM, it is not possible to "kill" a thread for an individual component without killing the entire JVM. 3. NiFi consists of well over 400 unique components and many of them are not executing native NiFi code. Many use client libraries not managed or controlled by NiFi. Others can be configured to execute command external to NiFi (ExecuteStreamCommand, ExecuteProcess, ExecuteScript, etc). Processors that invoke something external to NiFi's code base will result in a child process being created with its own pid. Keep in mind that processors of this type do not limit what externally is being invoked so take a generic approach to handling those child processes. The JVM invokes the external command and waits for it to respond complete. 4. Historically NiFi did not offer a terminate option since killing a thread in the NiFi JVM wis not possible. So when a component misbehaved (usually do to an issue external to NiFi code like network, client library hung, external command hung, etc), that NiFi component processor would get stuck just with the JVM thread waiting on that client library or external process to return. As such, the processor's concurrent task JVM thread is blocked. While you could select to stop the processor that would not help users get past the hung or long running thread. NiFi processors transition to a "stopping" state where it will remain until that library or task it is waiting on completes. Until that happens, users would not be able to modify the configuration or restart the component. This meant for truly hung issues the component would be blocked until the NiFi JVM was restarted. 5. As a result of the inconvenience/impact a hung thread causes, NiFi introduced the "terminate" option on a "stopped" component with an active thread. What Actually happens when a user clicks "terminate": 1. "Terminate" is only possible when after a processor has been asked to stop and that stopped processor still has associated JVM thread running. 2. Since we know that killing a JVM thread is not possible without killing the entire JVM process (NiFi), the "terminate" option takes a different approach. When a processor executes, it is doing so typically in response to inbound queued FlowFile as the trigger. That means the inbound FlowFile is tied to the JVM thread that is executing. When the thread completes, that FlowFiles (or modified, cloned, new FlowFile depending on processor function) is moved to the appropriate outbound relationship of the processor. 3. So what the "terminate" function really does is releases the FlowFile associated to that running JVM thread back to the inbound connection, makes request to client library or external command to abort/exit, and then isolates that thread so that if it does actually complete post terminate, all returns are just sent to null. 4. When "terminate" has been selected, the UI will render the processors active threads differently to indicate if the processor has JVM threads that have been terminated but are still active. NOTE: The number within the parenthesis denotes the current number of terminated threads still active. 5. If the client or external command responds to the request to exit, the active "terminated" thread will disappear. If not, it will continue to exist until thread finally completes or the entire NiFi JVM is restarted. NOTE: A terminated thread has little impact on resources since a hung thread isn't consuming CPU. Now a long running CPU intensive thread may have impact. 6. Now that this "terminate" JVM thread has been isolated and any FlowFile(s) tied to that thread have been released to originating connection, users can modify the processor configuration and start the component processor again. When started again, the processor will execute again on the FlowFile(s) that once belonged to the terminated thread. So no dataloss is incurred as a result of using "terminate". The "terminate" capability allows users to move on without needing to restart their NiFi JVM, thus reducing downtime and impact to other dataflows running on the NiFi canvas. If you have a processor that constantly has hung process issues or has very long running threads, it is time to start looking at your source FlowFile(s), processor configuration, external command, or external service the processor may be waiting for a response from as possible sources of the issue. Reference: Apache NiFi Terminate documentation
... View more
Labels:
10-03-2022
05:54 AM
1 Kudo
@leandrolinof I see no reason why using UpdateAttribute to establish the needed path and filename values for FetchSFTP processor would not work. FetchSFTP has no dependency on using ListSFTP. ListSFP just serves as a mechanism for obtaining a list of Files from a target SFTp server and recording state. ListSFTP simply creates a FlowFile with the needed attributes set for each File found on the target. So if you have another method build that can get set those attributes, then you are good to go. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more