Member since
07-30-2019
3472
Posts
1642
Kudos Received
1021
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 245 | 06-15-2026 08:08 AM | |
| 421 | 06-03-2026 06:06 PM | |
| 651 | 05-06-2026 09:16 AM | |
| 1481 | 05-04-2026 05:20 AM | |
| 749 | 05-01-2026 10:15 AM |
09-26-2023
07:19 AM
@lafi_oussama Have you tried upgrading to the latest version of Apache NiFi? You have not shared the specific ERROR you are encountering with the UnpackContent processor. You have not shared specific around what you tried using the ExecuteStreamCommand and Python scripts. Any additional details you can provide about your zip files? Thanks, Matt
... View more
09-22-2023
07:33 AM
@sid_21m Within NiFi, authentication and authorization are handled as separate processes. Upon successful authentication, NiFi has a user identity (In your case that user identity is your Azure AD username. That user identity is passed to the authorizer to make determination as to what NiFi authorization that user has been granted. At this point nothing more is known about the authenticated user other than the user identity. The Authorizer is configured in the authorizers.xml NiFi configuration file. In here you have multiple choices available to you, but none of them are capable of collecting App Roles from Azure. You can use the ldap-user-group-provider to collect ldap user to group associations from Azure AD. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
09-21-2023
01:14 PM
@edim2525 You can't have NiFi nodes in a cluster configured for different methods of authentication and authorization. User requests made on any node are proxied to the Cluster coordinator and then replicated to all nodes. It is unlikely that your user identity will remain the same once you change to using ldap for user authentication. Then you be setting up authorization based on those new user identities. Assuming you are currently using a managed-authorizer which uses the file-user-group-provider and file-access-policy-provider in your NiFi authorizers.xml? The ldap-provider can be configured to use the LDAP/AD DN (USE_DN) or the username entered at the login prompt (USE_USERNAME) as the user identity (case sensitive). Before making any changes to authentication, you could add the the new ldap based user identity in to your NiFi and authorize that user to all policies granted your current certificate based user already has. Then make a copy of the the login-identity-providers.xml file and Edit to add the the ldap-provider. Copy modified login-identity-providers.xml to all nodes. Then modify nifi.properties file on all nodes by changing following line: nifi.security.user.login.identity.provider=ldap-provider ***Theoretically (never done this) with authorization setup for your new ldap user identity setup across all nodes, you could probably restart one node at a time understanding that the only node that redirect to the new ldap-provider based login window would be a node that has been restarted. This way wok since your new ldap user identity will get proxied to the other nodes which will have authorization in place. On restart of your NiFi cluster these modified configuration files will be read. Keep in mind that when no other methods of authentication are enabled, NiFi will "REQUIRE" a client certificate for authentication through a mutualTLS exchange. Once additional methods of user authentication is added, mutualTLS auth is always enabled and attempted first, but instead of "REQUIRE", NiFI will "WANT" a client certificate. Only when no client certificate is presented during the MutualTLS exchange will NiFi move on to next configured method of authentication (ldap in your case). MutualTLS can NOT be disabled because it is only method of authentication for node to node communications. Now a caveat to above is that I have no idea about your current configuration, current user(s), how you plan to configure your ldap-provider, if you are using LDAP or AD, etc..., so guidance is very high level here. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
09-21-2023
12:22 PM
@Sivagopal What version of Apache NiFi are you using? There are some bugs that could lead to this. I recommend you are using Apache NiFi version 1.21 or newer. Is it always the same node that is disconnecting or appear random? Was the ERROR in the nifi-app.log followed by a full stack trace? - Make sure all nodes in your NiFi cluster are running with the same version of Apache NiFi. - Make sure that any custom or add-on nars and jars where added to all every node and have proper ownership and permissions. - Make sure all nodes use the same configured sensitive.props key and algorithm. Depending on your version of Apache NiFi, you may have a flow.xml.gz or both a flow.xml.gz and flow.json.gz files. On the node that is disconnecting, stop the node, and then rename the existing flow.xml.gz and flow.json.gz (if exists). If you restart NiFi, it will inherit the flow from the cluster when it joins and write new flow.xml.gz and flow.json.gz files to disk. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
09-20-2023
01:12 PM
@pacman NiFi has never supported security zones. There is no concept of zones in NiFi. NiFi's established policies are extremely granular all the way to the individual low level components (processors, controller services, etc) level. You have that level of individual user and group control within NiFi. NiFi also does not support "deny" ranger policies. A user is denied all except what they are authorized for directly or through inheritance. As far as to why /flow was working within a security zone... I have no idea. Would need to see that downloaded policy json and maybe I could figure out why??? Glad I was able to help you get going with your Ranger integration with NiFi. Matt
... View more
09-20-2023
12:50 PM
@pacman Are you sure NiFi is still successfully downloading the policy json when you make changes in Ranger? Within the ranger-nifi-security.xml file there is a configured "ranger.plugin.nifi.policy.cache" directory. Within that directory you will find the NiFi service .json policy file downloaded on the NiFi node. What is found in this file is what NiFi uses. Can you share that json file? Have you configured any security zones in Ranger (NiFi does not support this)? Have you unchecked the "delegate admin" check box on the policies in Ranger? Matt
... View more
09-20-2023
11:57 AM
Hello @pacman Thank s for sharing your authorizers.xml configuration. While you have both the file-user-group-provider and file-access-policy-provider configured in your authorizers.xml, your configured Ranger-provider does not use them. So these do nothing for you. When it comes to NiFi Resource Identifiers, they are very granular. The /flow NiFi Resource Identifier simply grants the authorized user access to the UI. It does not grant user ability to view (so you see no processor details or ability to open processor configuration to view it, modify, add, or delete any components. This is why everything on the canvas is ghosted (All users will always see ghosted components, it helps prevent different teams from building dataflows on top of one another). NiFi does not fully support wildcard in all NiFi Resource Identifiers, "/*" should work, but I would not recommend it. The intent here is to correctly and securely control these granular access controls. When you access NiFi's UI, you land in the root Process Group (NiFi builds this root process group for you when NiFi is first launched). Just like any other component a PG is assigned uuid. With nothing selected you will see the root PG uuid displayed in the operate panel to the left side of the canvas. Go ahead and copy that uuid. Go over to the Ranger and create a new policy in your NiFi Ranger service. In the NiFi Resource Identifier box paste that uuid. You should then be presented with available NiFi Resource Identifiers associated that PG. NiFi components inherit permissions from PGs. If grant user against a policy on that PG, all components created within that PG will inherit that same level of permissions unless you explicitly set a different policy on the sub component(s) or child process group(s). So if you give grant your user READ and Write on "/process-groups/<root PG uuid>", your user will be able to view and modify this process groups configuration and configuration of all components within this PG. Granting your user READ and WRITE on "/data/process-groups/<root pg uuid>" with allow you user to view content of FlowFile passed through components within this PG. The NiFi /resources endpoint returns all the available policies even those that would not be applicable when Ranger is managing authorization. So adding any authorizations related to NiFi Resource Identifiers that contain "policies" make no sense and are not necessary for example. I created community articles that maps the /NiFi Resource Identifiers used in Ranger to the NiFi user interface human readable policies you would have seen prior to switching to to Ranger as your authorizer (also describes what each grants the user). https://community.cloudera.com/t5/Community-Articles/NiFi-Ranger-based-policy-descriptions/ta-p/246586 https://community.hortonworks.com/articles/226382/nifi-restricted-components-policy-descriptions.html These should help you with your journey here. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
09-20-2023
11:22 AM
@need_help 100% agree with @cotopaul about using mergeRecord here. But I thought I would to explain what is happening with your current dataflow. The mergeContent processor is intended to be used to merge the content of multiple FlowFiles in to a single FlowFile. You have configured binary Concatenation which means append content of each FlowFile allocated to same bin to end of each other. So after merge you end up with 1 FlowFile with content from multiple source FlowFiles. So in your dataflow your ExecuteSQL is routing a lot of FlowFiles to the "failure" relationship all with same error. Your merge content when it is scheduled to execute will add FlowFiles to bin(s) based on configuration and then merge any bin that meets all minimums configured. So here you have min number of entries set to "1" and max number of entires set to "1000". That means that depending on how many FlowFiles are in the inbound failure connection to MergeContent before execution, you could have a bin with anywhere between 1 and 1000 FlowFiles allocated to it which then get merged resulting in one larger FlowFile with all that concatenated content. So the MergeContent is behaving exactly how you have it configured and output is expected. If you want a single email for every error, I am not clear on why you are using the mergeContent processor at all. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
09-20-2023
11:08 AM
@Frank37 Short answer is "No". There is nothing built within the NiDi core controller to accomplish exactly what you are trying to do. NiFi provides the ability to enable a long running task monitor which will alert to long running component threads, but that is all it does. This new feature was introduced in Apache NiFi 1.14 and disabled by default later in NIFI-8645 because it has an impact on performance because it needs to take thread dumps periodically and is only intended for troubleshooting. NiFi component processors are just pluggable components. NiFi core handles allocation of thread(s) to a component at component scheduled execution, but after a thread has been given to the processor, it is that processors code that must handle the lifespan of the thread. NiFi core would have no way of knowing how long a thread should or should not take to execute per every processor you add to the canvas. As you know from the linked article, the thread if truly hung will remain hung until NiFi is restarted, terminate will not kill a thread. So if NiFi was to imply an auto-terminate on all thread taking longer then X, that could be dangerous, Might end up terminating threads that actually do take longer, could end up with a continuously incrementing of hung threads by a misbehaving processor hat goes unnoticed, etc... Also once a thread has been terminated the NiFi core scheduler no longer cares about it, so starting processor again simply allows that processor to get a new thread from the cores Max Timer Driven thread pool. Best is to figure out why the threads are being hung and address it through processor configuration or code changes as needed. That being said, anything you can do through the UI can also be accomplished via the NiFi rest-api. This means you could possibly add a MonitorActivity processor after this custom processor that could use a lack of activity generated FlowFile to trigger a dataflow that makes rest-api calls to stop, terminate, and start the processor (but i would not recommend this approach for reason mentioned earlier). But it is not a simple dataflow (involves making request to change processor state stopped, then checking status of processor to see if it stopped or is stopping (where stopping indicates it is waiting on active threads), if it is "stopping" make a request to terminate thread, and finally restart processor). This also assumes a continuous flow fo FlowFiles through he processor. If it gets FlowFiles sporadicly, this would not work well. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
09-20-2023
10:36 AM
@pacman 1. There is noway to disable TLS. If you remove your TLS certificate from your browser or use and incognito window your client certificate will not be presented in the TLS exchange. NiFi requires TLS certificate authentication for NiFi to NiFi authenticated and authorized connections (for example in a multi-node NiFi cluster or utilizing NiFi Site-to-Site between different NiFi deployments). When NiFi is secured an NO other methods of authentication are configured, NiFi will "REQUIRE" a MutualTLS exchange. Once at least one additional method of authentication is configured, NiFi will "WANT" a client certificate and if one is not presented from the client, NiFi will move on to next authentication method. 2. I am not clear what you mean by "removed excessive permissions from authorizations.xml". If you are using Ranger, the authorizations.xml file is not being used. That file would have been created by the file-access-policy-provider. Ranger does not use this provider. There really is no concept of an "initial admin" when using Ranger. You'll need to add authorization for what you need manually in Ranger. The "Initial Admin" is used when NiFi authorization is handled by a local file provider so that a user can be setup on startup that has ability to access NiFi and setup additional authorizations from within the NiFi UI. I recommend starting a new community question so we don't make this thread overly complicated by solving many unrelated issues. There are some NiFi Resource Identifiers that would only apply to file based authorization, but all other do work when used correctly. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more