Member since
07-30-2019
3467
Posts
1641
Kudos Received
1018
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 149 | 05-06-2026 09:16 AM | |
| 245 | 05-04-2026 05:20 AM | |
| 236 | 05-01-2026 10:15 AM | |
| 467 | 03-23-2026 05:44 AM | |
| 352 | 02-18-2026 09:59 AM |
05-28-2021
09:23 AM
@Amn Where did you define the NiFi Attribute "${itemType}" (FlowFile Attribute on a source FlowFile driving execution of GetMongo processor or NiFi variable registry)? NiFi is also case sensitive, so for example "ItemType" and "itemType" would be considered different variables. If you found this helpful to yoru query, please take a moment to login and click "Accept" on this solution. Thank you, Matt
... View more
05-28-2021
09:14 AM
@Prajeesh10 If you are trying to split your source CSV in to two different FlowFile before converting each to a JSON, you could use the "SplitContent" [1] processor. [1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.SplitContent/index.html If you found this useful, please take a moment to login and click "Accept" on this solution. Thank you, Matt
... View more
05-28-2021
08:23 AM
@Rupesh_Raghani I just want to make sure that when we are both talking about NiFi "Templates" we are talking about the same thing. https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#templates When you upload a NiFi template (xml file) to NiFi via the UI (Does not matter which node in a NiFi cluster you are accessing), that template will get uploaded and replicated to all nodes in the cluster. So all nodes will have that template in each nodes JVM heap and written to the flow.xml.gz on disk. This statement is not clear to me: "So if I have templates on every node in cluster then will the nifi load templates on each node as per templates available on their node or a parent node will load all templates from other cluster". What do you mean by "if I have templates on every node"? In a NiFi cluster, every node must have the same flow.xml.gz. If the flow loaded in to heap memory does not match between nodes, the nodes no matching the elected cluster flow will be disconnected from the cluster. Each node, while it has its own local copy of the flow, runs the exact same flow. Hope this helps, Matt
... View more
05-27-2021
08:48 AM
@Rupesh_Raghani NiFI templates are uploaded to the process group selected. Only users who have been granted view and modify on that PG will be able to instantiate those templates on to the NiFi canvas. NiFi operates within a single JVM, so there are not separate per user canvases. User not authorized on a specific NiFi Process Group (PG) will only see a blank representation for that component on the canvas (non-authorized users can not see name, description, configuration details, etc...). Non-authorized users can still see metrics presented on components and queue counts on connections only. The reason NiFi still renders all components is because everything is executing within a single JVM. - One users flows can impact another team (Some flow with huge backlog of queued data, high volume, etc can impact resource availability to other teams/users. - Stats on information bar is representative of entire canvas (all dataflows), so it is still important for all teams/users to be able to identify areas of concern even if is not their dataflows and they have no authorizations allowing them to do anything about it. But they could alert other teams/users. - Typical use case is team/user 1 has a PG and then team/user 2 has a different authorized PG. If NiFi did not render both PG to both users, nothing prevents users from building flows on top of one another. Then assume we have team/user 3 that is authorized for both above team/users PGs. This becomes unreadable useable by that third user. If you found this helped with your query, please take a moment to login and click "Accept" on this solution. Thank you, Matt
... View more
05-27-2021
08:24 AM
@Rupesh_Raghani Since templates reside in NiFi heap, they should only be uploaded to the NiFi for the purpose of instantiating that template to the canvas. Once instantiate on to the canvas, the template should be deleted from the NiFi, so it is no longer holding that memory space. In addition to uploaded templates consuming heap memory space, so does everything build on the canvas (including controller services, and reporting tasks.). Additionally the metrics for each component also reside within heap memory space. Additionally, all FlowFiles queued (except large queues resulting in swap files) will also reside in the NiFi JVM heap memory space. How much heap each FlowFile consumes is driven by the number and size of the FlowFile attributes on each FlowFile (FlowFile content does not reside in heap memory except when a processor needs to do so to perform its task and not all processors need to touch the content at all and other may also read it without needing to hold it in heap if it is streaming it somewhere else.). The impact on heap various based on what components are being used and how many. If your flow grow extremely large, it may be a case of breaking those flows to be managed by Multiple NiFi clusters. NiFi flow templates will become a deprecated capability in favor of NiFi-Registry. You can version control your flows in to NiFi-Registry. All NiFi's connected to this NiFi-Registry can then load Flows from NiFi-Registry to the canvas (one or more times). I am not sure what you are looking for with regards to "How does it manage individual processors in memory?" All processors residing within the canvas and within templates will reside in the JVM heap memory space. If you find this helps with yoru query, please take a moment to login and click "ACCEPT" on this solution. Thank you, Matt
... View more
05-24-2021
06:24 AM
@Chakkara The more detail you can provide, the better we may be able to assist. Are you using custom processors? NiFi does not come with a ReplaceAttribute processor. I am assuming your flow may actually be using: GetFile (deprecated processor in favor of the listFile and FetchFile processors) --> UpdateAttribute --> ExtractText --> RouteOnAttribute --> putSQL Can you share the configuration you are currently using on the above processors, so we can see what you are doing now? Perhaps you only need a few tweaks to your current design. Thanks, Matt
... View more
05-21-2021
01:46 PM
@SAMSAL Sounds like you are very close... The 403 has nothing to do with authentication, but rather authorization. This eludes to the fact that the client was able to be properly trusted through the mutual TLS handshake, but when that resulting client string was checked against the endpoint policy being checked it was not present, thus resulting in a 403 response. The DN is the DistinquishedName for your certificate. You can use the below command to get the verbose details on the certs added to a keystore: keytool -v -list -keystore <keystore.jks or truststore.jks> You will see that each certificate has either type "PrivateKeyEntry" or "TrustedCertEntry" For each certificate you will see an "Owner" and an "Issuer" For each of those it will show the complete DN which would look something like: CN=<some string>, OU=<some string>, DC=<some string> There are numerous parts that can be part (CN, OU, O, ST, DC, etc.) of a DN and various lengths The "Owner" DN for the PrivateKeyEntry is what is used as the client/user authenticated string after the successful mutual TLS handshake. The nifi.properties file has optional ideNtity.mapping properties that can be configured to trim and manipulated these identity strings (for example pulling our on the string from the CN). With nifi.remote.input.secure= set to false, FlowFiles send over S2S will not be over a TLS encrypted connection. The fact that you say it works when you have this set to false and you can still successfully obtain S2S details from the configured https:// secured NiFi Destination URL tells me that authentication is correct for "Retrieve site-to-site details" NiFi policy. That only leaves having incorrect policy setup for your remote input port. You should open a command prompt on each node and "tail -F /<path to>/nifi-user.log". Then enable your controller services and check the logs being tailed for the authorization exception. It should clearly show you the "string" being checked against the "receive data via site-to-site" endpoint which in the log would look like "/data-transfer/input-ports/<uuid of remote input port>". Also remember that you are NOT authorizing your user to any of these S2S policies, but rather the DN or identity mapped DN string. Really hope this helps you get fully secured here. Matt
... View more
05-20-2021
03:25 PM
@karthikraja @Seedy Parameter Contexts are still relatively new. There is an existing improvement Jira for exactly what you are looking for here that is currently in development. NIFI-8487 Unfortunately, there is not workaround or alternative to what you are trying to achieve other than recreating those common parameters in every parameter context you create currently. If you found this query addressed your question, please take a moment to login and click accept on this solution. Thank you, Matt
... View more
05-20-2021
03:16 PM
@SAMSAL NiFi Site-To-SIte (S2S) components perform to things: 1. A background process runs every 30 seconds which connects to the target URL entered (In this case https://localhost:9443/nifi ) to retrieve S2S details. These details include details like how many nodes in target NiFi cluster, hostnames for those target NiFi nodes, load on those nodes, if those nodes support http and/or RAW transport protocols, What remote input ports exist that this source node is authorized to see, etc... S2S details are always fetched over HTTP even if you set transport protocol to RAW. 2. Then the source NiFi uses this data to actually sent content over S2S to all the target nifi nodes in a distributed fashion. Since your target is a https, the first thing that needs to happen is a Mutual TLS handshake. That means the keystore configured in the SSL Context service must contain a PrivateKeyEntry that support an EKU with "clientAuth". The target NiFi which is probably returned a FQDN via those S2S details, will also send its server certificate to the client. That means the truststore configured in your sslContextService must contain the complete trust chain for that certificate. That server certificate which comes from the keystore in the nifi.properties file on the target NiFi must contain a single PrivateKeyEntry with a EKU that supports "serverAuth" and must also have a SAN that matches the hostname used to connect with. The target NiFi's truststore configured in its nifi.properties file must also contain the complete trust chain for that client certificate presented from the sslContextService. So if all the above is properly in place, and I would guess it is since you are trying to S2S back to same NiFi cluster/instance and are probably using same keystore and truststore in your SSLContextService as is configured in the nifi.properties file, that those files are good. I would however be concerned with your use of "localhost" as the target URL because I doubt the server certificate sent in that TLS handshake is going to have a SAN entry that contains "localhost". You should instead provide the actual hostname for that target NiFi. It is fine and common to use localhost in the "instance URL"field as that is only used to identify the host that sent the FlowFile to the target InputPort. The only other statement that stands out to me is "The user created by securing the instance has the policy "retrieve site-to-site details"." NiFi authentication and authorization is setup to control what users are allowed to do once they access a secured NiFi UI. The components that are added by the authorized user do not execute as that authenticated user. All components execute as the NiFi service user. In that case of this S2S reporting task, it is executing as the NiFi service user but authenticating to the target through that mutual LS handshake, which means the DN form that clientAuth certificate is going to be the user that needs to be authorized for both the "retrieve Site-To-Site details" and "receive data via site-to-site" NiFi authorization policies. I know there is a lot of information here and hope it is clear. If you found this addressed your query, please take a moment to login and click accept on this solution. Thank you, Matt
... View more
05-20-2021
02:48 PM
1 Kudo
@leandrolinof NiFi Expression Language (NEL) [1] does not read content of the FlowFile. The RouteOnAttribute processor never looks at the content of a FlowFile. So verify your source FlowFile has attributes set with valid numeric only values. So your inbound FlowFile would need to have two attributes on it already: 1. cont 2. CONTADOR Note: NiFi is case sensitive as well. And both these attributes need to have assigned values to them. The NEL statement you have will return the value assigned to the FlowFile attribute "cont" and check to see if it is less than the value assigned to the FlowFile attribute "CONTADOR". If that resolves to "True", the FlowFile will be routed to the connection containing the new dynamically created "CONTINUE" relationship. Otherwise, it will route to the "unmatched" relationship which you appear to have auto-terminated. [1] https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html If you found this addressed your query, please take a moment to login and click accept on this solution. Thank you, Matt
... View more