Member since
07-30-2019
3373
Posts
1616
Kudos Received
998
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
24 | 10-20-2025 06:29 AM | |
164 | 10-10-2025 08:03 AM | |
144 | 10-08-2025 10:52 AM | |
136 | 10-08-2025 10:36 AM | |
201 | 10-03-2025 06:04 AM |
09-02-2025
05:44 AM
@yoonli Yes, your node certificate DNs do not need to be returned by your ldap. The common cluster setup I shared has both the file-user-group-provider (which you would use to define user identities manually) and ldap-user-group-provider (which syncs users and/or groups from your ldap so you don't need to add them manually). So the file-user-group-provider might look something like this in your authorizers.xml: <userGroupProvider>
<identifier>file-user-group-provider</identifier>
<class>org.apache.nifi.authorization.FileUserGroupProvider</class>
<property name="Users File">./conf/users.xml</property>
<property name="Legacy Authorized Users File"></property>
<property name="Initial User Identity 1">CN=nifi-cluster, OU=NiFi, O=NiFi Cluster, L=Hanoi, ST=HaNoi, C=VN</property>
<property name="Initial User Identity 2"></property>
</userGroupProvider> Above will create the user identity "CN=nifi-cluster, OU=NiFi, O=NiFi Cluster, L=Hanoi, ST=HaNoi, C=VN" for your node's certificate manually. it then becomes available for your file-access-policy provider to use to seed the node policies needed. If each of your NiFi nodes has a unique DN, you would add as many "Initial User Identity <num>" lines as needed to add them all on initial startup. ***REMINDER: users.xml and authorizations.xml files are ONLY created if they do not already exist. Edits to file-user-group-provider or file-access-policy provider will not edit preexisting files. The complete recommended authorizers.xml setup would look something like this: <authorizers>
<userGroupProvider>
<identifier>file-user-group-provider</identifier>
<class>org.apache.nifi.authorization.FileUserGroupProvider</class>
<property name="Users File">./conf/users.xml</property>
<property name="Legacy Authorized Users File"></property>
<property name="Initial User Identity 1">CN=nifi-cluster, OU=NiFi, O=NiFi Cluster, L=Hanoi, ST=HaNoi, C=VN</property>
<property name="Initial User Identity 2"></property>
</userGroupProvider>
<userGroupProvider>
<identifier>ldap-user-group-provider</identifier>
<class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class>
<property name="Authentication Strategy">ANONYMOUS</property>
<property name="Manager DN"></property>
<property name="Manager Password"></property>
<property name="TLS - Keystore"></property>
<property name="TLS - Keystore Password"></property>
<property name="TLS - Keystore Type"></property>
<property name="TLS - Truststore"></property>
<property name="TLS - Truststore Password"></property>
<property name="TLS - Truststore Type"></property>
<property name="TLS - Client Auth"></property>
<property name="TLS - Protocol"></property>
<property name="TLS - Shutdown Gracefully"></property>
<property name="Referral Strategy">FOLLOW</property>
<property name="Connect Timeout">10 secs</property>
<property name="Read Timeout">10 secs</property>
<property name="Url">ldap://localhost:10389</property>
<property name="Page Size"></property>
<property name="Sync Interval">30 mins</property>
<property name="Group Membership - Enforce Case Sensitivity">false</property>
<property name="User Search Base">ou=users,o=nifi</property>
<property name="User Object Class">person</property>
<property name="User Search Scope">ONE_LEVEL</property>
<property name="User Search Filter"></property>
<property name="User Identity Attribute">cn</property>
<property name="User Group Name Attribute"></property>
<property name="User Group Name Attribute - Referenced Group Attribute"></property>
<property name="Group Search Base">ou=groups,o=nifi</property>
<property name="Group Object Class">groupOfNames</property>
<property name="Group Search Scope">ONE_LEVEL</property>
<property name="Group Search Filter"></property>
<property name="Group Name Attribute">cn</property>
<property name="Group Member Attribute">member</property>
<property name="Group Member Attribute - Referenced User Attribute"></property>
</userGroupProvider>
<userGroupProvider>
<identifier>composite-user-group-provider</identifier>
<class>org.apache.nifi.authorization.CompositeConfigurableUserGroupProvider</class>
<property name="Configurable User Group Provider">file-user-group-provider</property>
<property name="User Group Provider 1">ldap-user-group-provider</property>
</userGroupProvider>
<accessPolicyProvider>
<identifier>file-access-policy-provider</identifier>
<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
<property name="User Group Provider">composite-user-group-provider</property>
<property name="Authorizations File">./conf/authorizations.xml</property>
<property name="Initial Admin Identity">nifi</property>
<property name="Legacy Authorized Users File"></property>
<property name="Node Identity 1">CN=nifi-cluster, OU=NiFi, O=NiFi Cluster, L=Hanoi, ST=HaNoi, C=VN/property>
<property name="Node Identity 2"></property>
</accessPolicyProvider>
<authorizer>
<identifier>managed-authorizer</identifier>
<class>org.apache.nifi.authorization.StandardManagedAuthorizer</class>
<property name="Access Policy Provider">file-access-policy-provider</property>
</authorizer>
</authorizers> *** Reminder: More then one user group provider added to the authorizers.xml can NOT return the same user or group identity. So do not add any user identities you are syncing from ldap-user-group-provider to the file-user-group-provider or NiFi will throw an exception complaining that two providers returned the same identity. Hope this helps clarify. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue(s) or answering your question(s), please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
08-29-2025
07:24 AM
@Virt_Apatt let me add some clarity to my last response with some example NiFi Expression Language (NEL) statements: NiFi attributes are stored as strings. In some cases NiFi will infer a number when it can, but when it comes to dates that is not possible. So when you create a FlowFile attribute with date string you'll need to convert to a date type before you can add you 10 days in milliseconds to it. Below is a NEL statement that you can enter your custom date into do the complete conversion in one step with out needing to pass the custom date in to the NEL from a FlowFile Attribute. ${literal('2023-01-15 10:30:00'):toDate('yyyy-MM-dd hh:mm:ss'):plus(864000000):format('yyyyMMddHHmmss')} Here is NEL statement where custom-Date comes from a pre-existing attribute on the FlowFile. This one allows you to get your initial "Custom Date" from a FlowFile attribute added to your FlowFile at some point upstream in your dataflow (pulling from some source, from some distributed cache, etc). This eliminates the need for the extra UpdateAttribute processor. ${'Custom Date':toDate('yyyy-MM-dd hh:mm:ss'):plus(864000000):format('yyyyMMddHHmmss')} Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
08-29-2025
06:34 AM
@yoonli Did you setup your authorizers.xml with cluster recommended structure i outlined earlier: From the log shared you can see following cause for your issue: Caused by: org.apache.nifi.authorization.exception.AuthorizerCreationException: org.apache.nifi.authorization.exception.AuthorizerCreationException: Unable to locate node CN=nifi-cluster, OU=NiFi, O=NiFi Cluster, L=Hanoi, ST=HaNoi, C=VN to seed policies. What this exception tells me is that you added your node's certificate DN to the file-access-policy-provider only. So on startup the file-access-policy-provider is attempting to seed NiFi nodes policies for that user identity, but it could not find that user identity because none of the configured user-group providers in the authorizers.xml created it. Since this user identity derived from your node's certificate DN is not being returned by your ldap-user-group-provider, it would need to be returned by the file-user-group-provider. For clarity, you are logging in just fine with your user. What is happening is your are logging into just one of yoru NiFi cluster nodes. So upon successful authentication the request to see the UI is sent to the elected cluster coordinator and replicated to all nodes in the cluster on your behalf. This is where the node is the proxy doing this for your successfully authenticated user. So you node is not authorized to proxy user requests. Replicating the user requests is required so changes are made all nodes in the cluster and so one node's UI can show data from all connected nodes. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue(s) or answering your question(s), please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
08-28-2025
06:13 AM
@yoonli That issue is unrelated to the issue here with getting your NiFi authorizer setup correctly. Glad you were able to get that resolved. If the feedback helped you resolve that issue, please accept that solution to help others in the community. What you are describing now is a missing authorization policy for your admin user. When NiFi's authorizer seeds the initial policies for the "admin" user, that does not mean the admin user has been given all access. It seeds the policies the admin user would need to access the UI and manage/modify authorizations for all users. The dataflow construction icon along the top of the UI are only visible to users who have been authorized on the Process Group (PG) being displayed. When NiFi is started for the first time it has no flow.json.gz file to load, so it generates one. That flow.json.gz will contain the root PG (NiFi default names the root PG "NiFi Flow"). Since your admin user is not authorized yet on this root PG, all the icons are greyed out. You'll also notice in the lower left corner or the UI and in the "Operation" panel on left side of canvas that the PG displays a UUID instead of the name "NiFi Flow". Now your admin user would have the authorization set to global manage authorization policies, so in that "Operations" panel you should see a key icon that is not greyed out. Clicking on that key will allow you admin to authorize users (including the admin user) to the authorizations polices specific to that PG: Select the Policy from the drop down you want to modify and you'll see a list of users and/or groups already authorized (I expect your's will be blank). After selecting the policy, you'll see an "Add users/groups to this policy" icon to the far right. Click on that and locate your admin user identity from the displayed list and click apply. View the component - will allow and authorized user to view details and configuration of a component. If that component is a process group, this authorization will be inherited by all components (processors, controller services, child process groups, etc) added to the canvas of that process group. But you have the ability to set authorizations explicitly on any component directly. So after setting this one you should start seeing "NiFi flow" PG name. Modify the component - will allow you to make changes. For a PG this means making all the construct icon to become available to you along the top of the UI. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
08-27-2025
07:23 AM
@Virt_Apatt Where is the "custom date" originating from. I see you are creating a FlowFile with a "Custom_Date" attribute which you then modify later, but is this custom date changing often? If so, where are you getting it from? Any dataflow that requires constant user updating is not going to be ideal. A more thorough and detailed use case might get you more responses in the community, especially when asking "is there a better way". Screenshot of your current dataflow? From what you shared, I don't see the need for the Update Attribute since you can manipulate the custom date directly in the GenerateFlowFile or ExecuteSQL processors by simply using the NiFi Expression Language (NEL) statement directly in during creation or during SQL query creation. Thanks, Matt
... View more
08-27-2025
06:00 AM
@vafs You shared two different authorizers.xml configurations (one that uses file-user-group-provider and another that uses the ldap-user-group-provider). Which one are you using when you experience your error? What version of Apache NiFi are you using? Is this a multi-node NiFi cluster or Single NiFi instance? Do you have an external load balance in front of your NiFi or are you accessing your NiFi instance(s) directly using the NiFi hosts assigned IP? While the configuration of the "ldap-provider" in the login-identity-providers.xml looks valid, the configuration in your "ldap-user-group-provider" in the authorizers.xml is not valid and is not going to return any user identities. The ldap-user-group-provider executes every 30 mins to resync users and maybe groups (depending on configuration). It executes independent of any outside input. So the following configuration line is not valid: <property name="User Search Filter">(uid={0})</property> The above is only valid in the "ldap-provider". The ldap-provider is triggered when a user supplies their username and password in the NiFi login prompt. "{0}" is then replaced with the username supplied at login. No such external input is passed to the ldap-user-group-provider, thus treating this search string as a literal value. In the ldap-user-group-provider you can leave the "User Search Filter" blank if your goal is to sync all users at ONE_LEVEL from your user search base: ou=users,dc=nifi,dc=local While the app.log does reveal some unauthorized request to the endpoint: https://10.29.144.56:8443/nifi-api/controller/nar-manager/nars, it it does not show the user identity that made the request. The nifi-user.log should include that information. The snippet you shared from both the nifi-app.log and nifi-user.log are not from the same time window. Additionally the snippet shared from the nifi-user.log is incomplete (often the very last "Caused by:" section in the full stack trace tells you what they initial failure was. keep in mind that the file-access-policy-provider is responsible for creating the authorizations.xml file ONLY when it does not already exist. When it needs to be created, it uses the file-access-policy-provider configuration to seed authorizations for an Initial Admin user identity and for your node identities (commonly retrieved from the file-user-group-provider, if a NiFi cluster setup is being used). Post existence of the authorizations.xml file, modifications are made to this file via admin user actions from directly in the NiFi UI. In a NiFi cluster setup the authorizer would common include the following providers: file-user-group-provider (contains only node identities derived from node certificates) ldap-user-group-provider (sync users and possibly groups) composite-configurable-user-group-provider (used to call both above providers) file-access-policy-provider (configured to use composite-configurable-user-group-provider) managed-authorizer There is not much other insights I can provide from the information provided. Hopefully this guidance will lead you to the useful log output and help you correct you configuration issue. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
08-26-2025
06:00 AM
@AlokKumar The out-of-the-box Single user setup is only intended to be used for ease of NiFi evaluation. For a multi user production NiFi setup you need to be using another form of multi user authentication and the managed authorizer for authorization. For authentication since you have only 5 developers you could go as simple as generated a unique user certificate for each of your 5 developers. These developers could load their certificate in to their browser and use that to authenticate their user during the mutual TKLS handshake between their browser and the NiFi instance. There are plenty of detailed how-tos via a google search on how to generate certificates and even some free services that will do it for you. You'll need to make sure the public certs are added to NiFi's truststore or NiFi will not trust the private certificates you create for your users. Another common option is using LDAP or Active Directory to authenticate your users. Again, there are many resource and examples on the web for setting up a LDAP server. LDAP is not a NiFi specific service. You can then follow the the NiFi guide links i shared in my previous post that show you what needs to be configured in NiFi to use LDAP. Client certificates are supported by any secured NiFi even if you have enabled additional form of authentication like LDAP. Client certificates are the only way NiFi nodes in a multi-node NIFi cluster setup authenticate with one another. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
08-26-2025
05:45 AM
@asand3r The question is how many partitions does he target Kafka topic have? If it only has 1 partition, then only one node in the consumeKafka consumer group is going to consume all the messages. Since you are saying that when LB on connection is disabled and queue shows all FlowFiles on one node, that tells me you have just one partition. For optimal throughput you would want some multiple of the number of nodes as the partition count. With 3 NiFi nodes, you would want 3, 6, 9, etc partitions. With 3 partitions and 1 concurrent task set on your consumeKafka, you will have 3 consumers in the consumer group. Each node will consume from one of those partitions. If you have 6 partitions, and 2 concurrent tasks set on the consumeKafka processor, you will have 6 consumers (3 nodes x 2 concurrent tasks) in your consumer group. So each node's consumeKafka will be able to concurrently pull from 2 partitions. So while your LB allows for redistribution of FlowFiles being consumed by one node, it is not the optimal setup here. LB connection will not change how the processor functions, it only operates against the FlowFiles output from the feeding processor. LB connection setting are most commonly used on downstream connection of processors that are typically scheduled on primary node only (listFile, ListSFTP, etc). Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
08-25-2025
12:57 PM
@GKHN_ As I described in my first response, Authentication and Authorization are two different processes. So it sounds like from your comment that authentication is working fine for both your users and authorization is failing for your non admin user. So issue is within the authorization phase. I assume both of your users are authenticating via ldap? In your ldap-provider in the login-identity-providers.xml you have the "Identity Strategy" set to "USE_DN". With this setting you the users full ldap DN will be used as the user identity string after successful authentication. This means that entire DN is being passed to the authorizer to lookup if that full dn has been authorized to the requested end-point NiFi policy. I see you have your initial admin identity manually defined in the file-user-group-provider and the file-access-policy provider: CN=NIFIUSER,OU=Userpro,OU=CUsers,OU=Company,DC=company,DC=entp So when you login via ldap with this user's ldap username and ldap password, the user's entire DN is being passed to the authorizer and the file-access-policy provider has setup all admin related NiFi policies for this initial admin user identity. I also see from the shared authorizers.xml that the only user-group-provider the "file-access-policy provider" is configured to use is the "file-user-group-provider". The file-user-group-provider requires the admin user to manually add additional user identities manually from the with the NiFi UI (Remember that with your current ldap-provider login provider, all your ldap user identities are going to be full DNs). As the admin user, go to the NiFi global menu and select "USERS": From the NiFi Users UI, select the "+" to add a new user: Then enter the full DN for your second user (Case sensitive). unless you have added any groups, your list of groups will be blank. Now that you have added this second user identity, you'll need to start authorizing that user identities for the various policy they need. In order to access the NiFi UI, all users must be authorized to "view the user interface". From the same NiFi Global menu mentioned above, select "Policies" this time. Then from the "Access Policies" UI that appears, select "view the user interface" from the policy list pull-down. Then click on the icon to the right that looks like a person with a "+". Find the user identity you just added and check the box and click the "Add" button. Now this user can access the NIFi UI. There are other policies this user will need before they can start building dataflows on the UI. NiFi allows for very granular authorizations. But at the minimum the user will need to be authorized on the process group in which they will build their dataflows. Not all policies are defined from the "Access Policies" UI in the global menu. the component level policies are define directly via the individual component (keep an eye out for the "key" icon) From the "Operation" panel directly on the NiFi canvas you can set policies on the currently selected component: Above I have selected my root Process Group (PG). If you click the key icon you will see all the access policies that users can be authorized for. You'll need to select each one by one your user will need and add the user to them. Above will allow you to setup access for your additional users using the file-user-group-provider you have configured in your authorizers.xml. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
08-25-2025
08:21 AM
1 Kudo
@Amry The "clear state" option only appears if ListSFTP is configured to use "tracking timestamps" and there is state available to be cleared. Since you are using "tracking entities", entities are being tracked in a NiFi MapCacheServer. If your MapCacheServer has been configured with a "persistence Directory", you'll need to stop the cache server, delete the contents of the persistence directory and then restart the cache server to clear the data. If your MapCacheServer has NOT been configured with a "persistence Directory", you'll need to stop the cache server and then restart the cache server to clear the data. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more