@yoonli  Yes, your node certificate DNs do not need to be returned by your ldap. The common cluster setup I shared has both the file-user-group-provider (which you would use to define user identities manually) and ldap-user-group-provider (which syncs users and/or groups from your ldap so you don't need to add them manually). So the file-user-group-provider might look something like this in your authorizers.xml: <userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.authorization.FileUserGroupProvider</class> <property name="Users File">./conf/users.xml</property> <property name="Legacy Authorized Users File"></property> <property name="Initial User Identity 1">CN=nifi-cluster, OU=NiFi, O=NiFi Cluster, L=Hanoi, ST=HaNoi, C=VN</property> <property name="Initial User Identity 2"></property> </userGroupProvider> Above will create the user identity "CN=nifi-cluster, OU=NiFi, O=NiFi Cluster, L=Hanoi, ST=HaNoi, C=VN" for your node's certificate manually.  it then becomes available for your file-access-policy provider to use to seed the node policies needed.  If each of your NiFi nodes has a unique DN, you would add as many "Initial User Identity <num>" lines as needed to add them all on initial startup.  ***REMINDER: users.xml and authorizations.xml files are ONLY created if they do not already exist.  Edits to file-user-group-provider or file-access-policy provider will not edit preexisting files. The complete  recommended authorizers.xml setup would look something like this: <authorizers> <userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.authorization.FileUserGroupProvider</class> <property name="Users File">./conf/users.xml</property> <property name="Legacy Authorized Users File"></property> <property name="Initial User Identity 1">CN=nifi-cluster, OU=NiFi, O=NiFi Cluster, L=Hanoi, ST=HaNoi, C=VN</property> <property name="Initial User Identity 2"></property> </userGroupProvider> <userGroupProvider> <identifier>ldap-user-group-provider</identifier> <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class> <property name="Authentication Strategy">ANONYMOUS</property> <property name="Manager DN"></property> <property name="Manager Password"></property> <property name="TLS - Keystore"></property> <property name="TLS - Keystore Password"></property> <property name="TLS - Keystore Type"></property> <property name="TLS - Truststore"></property> <property name="TLS - Truststore Password"></property> <property name="TLS - Truststore Type"></property> <property name="TLS - Client Auth"></property> <property name="TLS - Protocol"></property> <property name="TLS - Shutdown Gracefully"></property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url">ldap://localhost:10389</property> <property name="Page Size"></property> <property name="Sync Interval">30 mins</property> <property name="Group Membership - Enforce Case Sensitivity">false</property> <property name="User Search Base">ou=users,o=nifi</property> <property name="User Object Class">person</property> <property name="User Search Scope">ONE_LEVEL</property> <property name="User Search Filter"></property> <property name="User Identity Attribute">cn</property> <property name="User Group Name Attribute"></property> <property name="User Group Name Attribute - Referenced Group Attribute"></property> <property name="Group Search Base">ou=groups,o=nifi</property> <property name="Group Object Class">groupOfNames</property> <property name="Group Search Scope">ONE_LEVEL</property> <property name="Group Search Filter"></property> <property name="Group Name Attribute">cn</property> <property name="Group Member Attribute">member</property> <property name="Group Member Attribute - Referenced User Attribute"></property> </userGroupProvider> <userGroupProvider> <identifier>composite-user-group-provider</identifier> <class>org.apache.nifi.authorization.CompositeConfigurableUserGroupProvider</class> <property name="Configurable User Group Provider">file-user-group-provider</property> <property name="User Group Provider 1">ldap-user-group-provider</property> </userGroupProvider> <accessPolicyProvider> <identifier>file-access-policy-provider</identifier> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class> <property name="User Group Provider">composite-user-group-provider</property> <property name="Authorizations File">./conf/authorizations.xml</property> <property name="Initial Admin Identity">nifi</property> <property name="Legacy Authorized Users File"></property> <property name="Node Identity 1">CN=nifi-cluster, OU=NiFi, O=NiFi Cluster, L=Hanoi, ST=HaNoi, C=VN/property> <property name="Node Identity 2"></property> </accessPolicyProvider> <authorizer> <identifier>managed-authorizer</identifier> <class>org.apache.nifi.authorization.StandardManagedAuthorizer</class> <property name="Access Policy Provider">file-access-policy-provider</property> </authorizer> </authorizers> *** Reminder: More then one user group provider added to the authorizers.xml can NOT return the same user or group identity.  So do not add any user identities you are syncing from ldap-user-group-provider to the file-user-group-provider or NiFi will throw an exception complaining that two providers returned the same identity. Hope this helps clarify.   Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue(s) or answering your question(s), please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Virt_Apatt  Where is the "custom date" originating from.  I see you are creating a FlowFile with a "Custom_Date" attribute which you then modify later, but is this custom date changing often?  If so, where are you getting it from? Any dataflow that requires constant user updating is not going to be ideal.   A more thorough and detailed use case might get you more responses in the community, especially when asking "is there a better way".   Screenshot of your current dataflow? From what you shared, I don't see the need for the Update Attribute since you can manipulate the custom date directly in the GenerateFlowFile or ExecuteSQL processors by simply using the NiFi Expression Language (NEL) statement directly in during creation or during SQL query creation. Thanks, Matt ... View more

@vafs  You shared two different authorizers.xml configurations (one that uses file-user-group-provider and another that uses the ldap-user-group-provider).  Which one are you using when you experience your error? What version of Apache NiFi are you using? Is this a multi-node NiFi cluster or Single NiFi instance? Do you have an external load balance in front of your NiFi or are you accessing your NiFi instance(s) directly using the NiFi hosts assigned IP?  While the configuration of the "ldap-provider" in the login-identity-providers.xml looks valid, the configuration in your "ldap-user-group-provider" in the authorizers.xml is not valid and is not going to return any user identities. The ldap-user-group-provider executes every 30 mins to resync users and maybe groups (depending on configuration).  It executes independent of any outside input.  So the following configuration line is not valid: <property name="User Search Filter">(uid={0})</property> The above is only valid in the "ldap-provider".  The ldap-provider is triggered when a user supplies their username and password in the NiFi login prompt. "{0}" is then replaced with the username supplied at login.  No such external input is passed to the ldap-user-group-provider, thus treating this search string as a literal value. In the ldap-user-group-provider you can leave the "User Search Filter" blank if your goal is to sync all users at ONE_LEVEL from your user search base: ou=users,dc=nifi,dc=local While the app.log does reveal some unauthorized request to the endpoint: https://10.29.144.56:8443/nifi-api/controller/nar-manager/nars, it it does not show the user identity that made the request.  The nifi-user.log should include that information.  The snippet you shared  from both the nifi-app.log and nifi-user.log are not from the same time window.  Additionally the snippet shared from the nifi-user.log is incomplete (often the very last "Caused by:" section in the full stack trace tells you what they initial failure was. keep in mind that the file-access-policy-provider is responsible for creating the authorizations.xml file ONLY when it does not already exist.  When it needs to be created, it uses the file-access-policy-provider configuration to seed authorizations for an Initial Admin user identity and for your node identities (commonly retrieved from the file-user-group-provider, if a NiFi cluster setup is being used).  Post existence of the authorizations.xml file, modifications are made to this file via admin user actions from directly in the NiFi UI. In a NiFi cluster setup the authorizer would common include the following providers: file-user-group-provider (contains only node identities derived from node certificates) ldap-user-group-provider (sync users and possibly groups) composite-configurable-user-group-provider (used to call both above providers) file-access-policy-provider (configured to use composite-configurable-user-group-provider) managed-authorizer  There is not much other insights I can provide from the information provided.  Hopefully this guidance will lead you to the useful log output and help you correct you configuration issue. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more

@AlokKumar  The out-of-the-box Single user setup is only intended to be used for ease of NiFi evaluation.  For a multi user production NiFi setup you need to be using another form of multi user authentication and the managed authorizer for authorization.   For authentication since you have only 5 developers you could go as simple as generated a unique user certificate for each of your 5 developers.  These developers could load their certificate in to their browser and use that to authenticate their user during the mutual TKLS handshake between their browser and the NiFi instance.   There are plenty of detailed how-tos via  a google  search on how to generate certificates and even some free services that will do it for you.   You'll need to make sure the public certs are added to NiFi's truststore or NiFi will not trust the private certificates you create for your users. Another common option is using LDAP or Active Directory to authenticate your users.  Again, there are many resource and examples on the web for setting up a LDAP server.  LDAP is not a NiFi specific service.   You can then follow the the NiFi guide links i shared in my previous post that show you what needs to be configured in NiFi to use LDAP.    Client certificates are supported by any secured NiFi even if you have enabled additional form of authentication like LDAP.  Client certificates are the only way NiFi nodes in a multi-node NIFi cluster setup authenticate with one another.   Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@asand3r  The question is how many partitions does he target Kafka topic have? If it only has 1 partition, then only one node in the consumeKafka consumer group is going to consume all the messages.  Since you are saying that when LB on connection is disabled and queue shows all FlowFiles on one node, that tells me you have just one partition.  For optimal throughput you would want some multiple of the number of nodes as the partition count. With 3 NiFi nodes, you would want 3, 6, 9, etc partitions.  With 3 partitions and 1 concurrent task set on your consumeKafka, you will have 3 consumers in the consumer group.  Each node will consume  from one of those partitions.  If you have 6 partitions, and 2 concurrent tasks set on the consumeKafka processor, you will have 6 consumers (3 nodes x 2 concurrent tasks) in  your consumer group.  So each node's consumeKafka will be able to concurrently pull from 2 partitions.   So while your LB allows for redistribution of FlowFiles being consumed by one node, it is not the optimal setup here.  LB connection will not change how the processor functions, it only operates against the FlowFiles output from the feeding processor.  LB connection setting are most commonly used on downstream connection of processors that are typically scheduled on primary node only (listFile, ListSFTP, etc). Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@GKHN_  As I described in my first response, Authentication and Authorization are two different processes.   So it sounds like from your comment that authentication is working fine for both your users and authorization is failing for your non admin user.  So issue is within the authorization phase. I assume both of your users are authenticating via ldap? In your ldap-provider in the login-identity-providers.xml you have the "Identity Strategy" set to  "USE_DN".   With this setting you the users full ldap DN will be used as the user identity string after successful authentication.  This means that entire DN is being passed to the authorizer to lookup if that full dn has been authorized to the requested end-point NiFi policy. I see you have your initial admin identity manually defined in the file-user-group-provider and the file-access-policy provider: CN=NIFIUSER,OU=Userpro,OU=CUsers,OU=Company,DC=company,DC=entp So when you login via ldap with this user's ldap username and ldap password, the user's entire DN is being passed to the authorizer and the file-access-policy provider has setup all admin related NiFi policies for this initial admin user identity. I also see from the shared authorizers.xml that the only user-group-provider the "file-access-policy provider" is configured to use is the "file-user-group-provider".  The file-user-group-provider requires the admin user to manually add additional user identities manually from the with the NiFi UI (Remember that with your current ldap-provider login provider, all your ldap user identities are going to be full DNs). As the admin user, go to the NiFi global menu and select "USERS": From the NiFi Users UI, select the "+" to add a new user: Then enter the full DN for your second user (Case sensitive). unless you have added any groups, your list of groups will be blank. Now that you have added this second user identity, you'll need to start authorizing that user identities for the various policy they need.  In order to access the NiFi UI, all users must be authorized to "view the user interface".  From the same NiFi Global menu mentioned above, select "Policies" this time.  Then from the "Access Policies" UI that appears, select "view the user interface" from the policy list pull-down. Then click on the icon to the right that looks like a person with a "+".  Find the user identity you just added and check the box and click the "Add" button. Now this user can access the NIFi UI. There are other policies this user will need before they can start building dataflows on the UI.  NiFi allows for very granular authorizations.   But at the minimum the user will need to be authorized on the process group in which they will build their dataflows.  Not all policies are defined from  the "Access Policies" UI in the global menu. the component level policies are define directly via the individual component (keep an eye out for the "key" icon)  From the "Operation" panel directly on the NiFi canvas you can set policies on the currently selected component: Above I have selected my root Process Group (PG). If you click the key icon you will see all the access policies that users can be authorized for.  You'll need to select each one by one your user will need and add the user to them. Above will allow you to setup access for your additional users using the file-user-group-provider you have configured in your authorizers.xml. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more