Member since
07-30-2019
3470
Posts
1642
Kudos Received
1018
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 262 | 05-06-2026 09:16 AM | |
| 451 | 05-04-2026 05:20 AM | |
| 319 | 05-01-2026 10:15 AM | |
| 513 | 03-23-2026 05:44 AM | |
| 388 | 02-18-2026 09:59 AM |
09-21-2023
12:22 PM
@Sivagopal What version of Apache NiFi are you using? There are some bugs that could lead to this. I recommend you are using Apache NiFi version 1.21 or newer. Is it always the same node that is disconnecting or appear random? Was the ERROR in the nifi-app.log followed by a full stack trace? - Make sure all nodes in your NiFi cluster are running with the same version of Apache NiFi. - Make sure that any custom or add-on nars and jars where added to all every node and have proper ownership and permissions. - Make sure all nodes use the same configured sensitive.props key and algorithm. Depending on your version of Apache NiFi, you may have a flow.xml.gz or both a flow.xml.gz and flow.json.gz files. On the node that is disconnecting, stop the node, and then rename the existing flow.xml.gz and flow.json.gz (if exists). If you restart NiFi, it will inherit the flow from the cluster when it joins and write new flow.xml.gz and flow.json.gz files to disk. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
09-20-2023
01:12 PM
@pacman NiFi has never supported security zones. There is no concept of zones in NiFi. NiFi's established policies are extremely granular all the way to the individual low level components (processors, controller services, etc) level. You have that level of individual user and group control within NiFi. NiFi also does not support "deny" ranger policies. A user is denied all except what they are authorized for directly or through inheritance. As far as to why /flow was working within a security zone... I have no idea. Would need to see that downloaded policy json and maybe I could figure out why??? Glad I was able to help you get going with your Ranger integration with NiFi. Matt
... View more
09-20-2023
12:50 PM
@pacman Are you sure NiFi is still successfully downloading the policy json when you make changes in Ranger? Within the ranger-nifi-security.xml file there is a configured "ranger.plugin.nifi.policy.cache" directory. Within that directory you will find the NiFi service .json policy file downloaded on the NiFi node. What is found in this file is what NiFi uses. Can you share that json file? Have you configured any security zones in Ranger (NiFi does not support this)? Have you unchecked the "delegate admin" check box on the policies in Ranger? Matt
... View more
09-20-2023
11:57 AM
Hello @pacman Thank s for sharing your authorizers.xml configuration. While you have both the file-user-group-provider and file-access-policy-provider configured in your authorizers.xml, your configured Ranger-provider does not use them. So these do nothing for you. When it comes to NiFi Resource Identifiers, they are very granular. The /flow NiFi Resource Identifier simply grants the authorized user access to the UI. It does not grant user ability to view (so you see no processor details or ability to open processor configuration to view it, modify, add, or delete any components. This is why everything on the canvas is ghosted (All users will always see ghosted components, it helps prevent different teams from building dataflows on top of one another). NiFi does not fully support wildcard in all NiFi Resource Identifiers, "/*" should work, but I would not recommend it. The intent here is to correctly and securely control these granular access controls. When you access NiFi's UI, you land in the root Process Group (NiFi builds this root process group for you when NiFi is first launched). Just like any other component a PG is assigned uuid. With nothing selected you will see the root PG uuid displayed in the operate panel to the left side of the canvas. Go ahead and copy that uuid. Go over to the Ranger and create a new policy in your NiFi Ranger service. In the NiFi Resource Identifier box paste that uuid. You should then be presented with available NiFi Resource Identifiers associated that PG. NiFi components inherit permissions from PGs. If grant user against a policy on that PG, all components created within that PG will inherit that same level of permissions unless you explicitly set a different policy on the sub component(s) or child process group(s). So if you give grant your user READ and Write on "/process-groups/<root PG uuid>", your user will be able to view and modify this process groups configuration and configuration of all components within this PG. Granting your user READ and WRITE on "/data/process-groups/<root pg uuid>" with allow you user to view content of FlowFile passed through components within this PG. The NiFi /resources endpoint returns all the available policies even those that would not be applicable when Ranger is managing authorization. So adding any authorizations related to NiFi Resource Identifiers that contain "policies" make no sense and are not necessary for example. I created community articles that maps the /NiFi Resource Identifiers used in Ranger to the NiFi user interface human readable policies you would have seen prior to switching to to Ranger as your authorizer (also describes what each grants the user). https://community.cloudera.com/t5/Community-Articles/NiFi-Ranger-based-policy-descriptions/ta-p/246586 https://community.hortonworks.com/articles/226382/nifi-restricted-components-policy-descriptions.html These should help you with your journey here. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
09-20-2023
11:22 AM
@need_help 100% agree with @cotopaul about using mergeRecord here. But I thought I would to explain what is happening with your current dataflow. The mergeContent processor is intended to be used to merge the content of multiple FlowFiles in to a single FlowFile. You have configured binary Concatenation which means append content of each FlowFile allocated to same bin to end of each other. So after merge you end up with 1 FlowFile with content from multiple source FlowFiles. So in your dataflow your ExecuteSQL is routing a lot of FlowFiles to the "failure" relationship all with same error. Your merge content when it is scheduled to execute will add FlowFiles to bin(s) based on configuration and then merge any bin that meets all minimums configured. So here you have min number of entries set to "1" and max number of entires set to "1000". That means that depending on how many FlowFiles are in the inbound failure connection to MergeContent before execution, you could have a bin with anywhere between 1 and 1000 FlowFiles allocated to it which then get merged resulting in one larger FlowFile with all that concatenated content. So the MergeContent is behaving exactly how you have it configured and output is expected. If you want a single email for every error, I am not clear on why you are using the mergeContent processor at all. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
09-20-2023
11:08 AM
@Frank37 Short answer is "No". There is nothing built within the NiDi core controller to accomplish exactly what you are trying to do. NiFi provides the ability to enable a long running task monitor which will alert to long running component threads, but that is all it does. This new feature was introduced in Apache NiFi 1.14 and disabled by default later in NIFI-8645 because it has an impact on performance because it needs to take thread dumps periodically and is only intended for troubleshooting. NiFi component processors are just pluggable components. NiFi core handles allocation of thread(s) to a component at component scheduled execution, but after a thread has been given to the processor, it is that processors code that must handle the lifespan of the thread. NiFi core would have no way of knowing how long a thread should or should not take to execute per every processor you add to the canvas. As you know from the linked article, the thread if truly hung will remain hung until NiFi is restarted, terminate will not kill a thread. So if NiFi was to imply an auto-terminate on all thread taking longer then X, that could be dangerous, Might end up terminating threads that actually do take longer, could end up with a continuously incrementing of hung threads by a misbehaving processor hat goes unnoticed, etc... Also once a thread has been terminated the NiFi core scheduler no longer cares about it, so starting processor again simply allows that processor to get a new thread from the cores Max Timer Driven thread pool. Best is to figure out why the threads are being hung and address it through processor configuration or code changes as needed. That being said, anything you can do through the UI can also be accomplished via the NiFi rest-api. This means you could possibly add a MonitorActivity processor after this custom processor that could use a lack of activity generated FlowFile to trigger a dataflow that makes rest-api calls to stop, terminate, and start the processor (but i would not recommend this approach for reason mentioned earlier). But it is not a simple dataflow (involves making request to change processor state stopped, then checking status of processor to see if it stopped or is stopping (where stopping indicates it is waiting on active threads), if it is "stopping" make a request to terminate thread, and finally restart processor). This also assumes a continuous flow fo FlowFiles through he processor. If it gets FlowFiles sporadicly, this would not work well. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
09-20-2023
10:36 AM
@pacman 1. There is noway to disable TLS. If you remove your TLS certificate from your browser or use and incognito window your client certificate will not be presented in the TLS exchange. NiFi requires TLS certificate authentication for NiFi to NiFi authenticated and authorized connections (for example in a multi-node NiFi cluster or utilizing NiFi Site-to-Site between different NiFi deployments). When NiFi is secured an NO other methods of authentication are configured, NiFi will "REQUIRE" a MutualTLS exchange. Once at least one additional method of authentication is configured, NiFi will "WANT" a client certificate and if one is not presented from the client, NiFi will move on to next authentication method. 2. I am not clear what you mean by "removed excessive permissions from authorizations.xml". If you are using Ranger, the authorizations.xml file is not being used. That file would have been created by the file-access-policy-provider. Ranger does not use this provider. There really is no concept of an "initial admin" when using Ranger. You'll need to add authorization for what you need manually in Ranger. The "Initial Admin" is used when NiFi authorization is handled by a local file provider so that a user can be setup on startup that has ability to access NiFi and setup additional authorizations from within the NiFi UI. I recommend starting a new community question so we don't make this thread overly complicated by solving many unrelated issues. There are some NiFi Resource Identifiers that would only apply to file based authorization, but all other do work when used correctly. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
09-19-2023
07:15 AM
@edim2525 If your NiFi is already secured that means you have already using authentication and authorization is some form of configuration. So depending in your current secure setup configuration, the guidance you may need will vary. There are multiple NiFi configuration files that establish the configurations for authentication and authorization (While authorization is dependent on successful authentication, the processes are executed separately). nifi.properties login-identity-providers.xml authorizers.xml Understanding your current setup is important for giving proper guidance to change configuration. For authenticating with LDAP/AD users, you'll want to use the ldap-provider in the login-identity-providers.xml For authorization you can NOT use default "single-user-authorizer" in the authorizers.xml authorizers.xml. You'll need to switch to a different provider like the Standard Managed Authorizer. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
09-19-2023
07:00 AM
1 Kudo
@SandyClouds The "Jolt Specification" property in the JoltTransformJson processor supports NiFi Expression Language (will be evaluated using flow file attributes and variable registry). This means that any attribute present on the FlowFile passed to the JoltTransform processor can be used in the spec. So if you have an upstream processor adding an attribute to your FlowFile with the attribute name "customer_id", you can simply use NiFi Expression Language in your spec to pull in that attributes value. NiFi Parameters are defined using "#{<parameter name>}" which will look up that parameter in the parameter context defined on the NiFi Process Group and return its value. NiFi Expression language is defined by starting with "${" and ending with "}". Between that you can bring in an attributes value, apply NiFi Expression Language functions against an attributes value, etc. Simplest NiFi Expression Language statement would be "${customer_id}", which will look for an attribute in the FlowFile being processed and return its value. We refer to "customer_id" as the "subject" in that NiFi EL. NiFi EL does have a bunch of subjectless functions like "now()" that you are using, but more typical use cases start with a subject (FlowFile attribute). I strongly encourage you to read through the linked NiFi EL guide to help you better understand the complexity and many ways you can used NiFi EL statements. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
09-19-2023
05:59 AM
@pacman The "unable to find valid certificate path to request target" is caused when trust can not be established in the TLS handshake. This is the result of missing trusted cert entry(s) in the truststore (missing or incomplete trust chain). When you used the toolkit to create your certficates, it also built the corresponding truststore with the required trust chain. The HTTP 403 exception indicated not authorized while the 401 indicated failed authentication. Authentication was determined through the mutualTLS handshake between the clientAuth certificate presented from the keystore configured in Ranger and serverAuth certificate presented by NiFi. The DN from the clientAuth certificate is used as the user identity in the authentication (keep in mind that during authentication the final user identity is determined after evaluating the user identity returned during authentication against any configured Identity mapping patterns configured in the nifi.properties file. If you inspect the nifi-user.log, you should see attempt to access the .../resources NiFi endpoint (This is endpoint where Ranger is trying to fetch the list of NiFi resource identifiers). In that nifi-user.log you will see the actual post authentication user identity being used (full certificate DN or some valued mapped from that full DN). This is where my early response about needing to authorize the ranger clientAuth certificate user identity in Ranger manually against the "/resources" NiFi resource Identifier is needed. So you'll need to make sure the exact user identity from the nifi-user.log is added as a user in Ranger and added to a NiFi service policy in ranger giving that user authorization for "/resources". Once NiFi downloads the updated policy json from Ranger, Ranger NiFi service client will now be authorized to fetch all NiFi Resource Identifiers from the /resources endpoint. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more