@JobBranwl  So your ldap-user-group-provider is returning the value from the "cn" attribute which is likely just: initial-admin  However, you set your initial admin identity in the file-access-policy provider to a full DN.  So when that provider attempts to build the authorization.xml, it can't because neither provider returned that identity string. I recommend you leave Identity strategy as "USE_USERNAME" (most commonly used). I also recommend setting your initial admin identity in the file-access-policy provider to your admin username and not the full DN.  You can also add a logger to the NiFi logback.xml that will result in all the user and group identities strings being output in the nifi-app.log for inspection by you to aid in yoru troubleshooting here:  <logger name="org.apache.nifi.ldap.tenants.LdapUserGroupProvider" level="DEBUG"/>  Just scroll down in your logback until you start seeing logger lines and add this one. User and groups produced by the user-group providers are assigned a uuid.  That uuid is then used when assigning policies in the authorization.xml. That is why a user-group provider must return an identity in order to set permissions or seed initial policies against a user.  That is why you got that exception. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@edaley  I also recommend upgrading to latest NiFi release to see if this issue persists.  I believe there was a known scheduling on startup bug in Apache NiFi 1.16 that since has been resolved. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@edaley  When you NiFi cluster is in this state: 1. How many active threads are indicated in the UI? 2. On which processors are these threads active? (NiFi Summary UI --> processors) Number in parenthesis is number of active threads on that component. 3. Does either of above stats change or same processors continue to show active threads with no progress of FlowFiles through those processors? 4. Does number of active threads match the size of your thread pool (global menu --> GENERAL --> "Maximum Timer Driven Thread Count)?  This setting is per node, so default is 10 and if you have 3 nodes that could mean you see 30 on the UI as active. 4. Have you use NiFi.sh (./nifi.sh dump <dumpfilename>) to produce several thread dumps to see if any of those threads are changing between dumps (change indicates thread is processing.  No change points at potentially hung thread of a thread waiting on another long running or hung thread. Hopefully this information will help you troubleshoot and narrow down your issue.   If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@ajignacio  In the stack trace yoou'll see below: Caused by: org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force   This indicates that something external to your NiFi is changing the provenance files.  When we see this it is most commonly the result of some virus software scanning the NiFi repositories and during that modifying the the files.    You should make sure that any external service exclude the NiFi repositories from scanning and modification. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@JobBranwl  Within your authorizers.xml file you can NOT have multiple user-group providers return the same user identity string. I see you creating the following user identity string in the file-user-group-provider   cn=Myuser,ou=Myuser,dc=Mydomain,dc=com    and also see in your ldap-user-group-provider that you may be trying to sync the sme user from your LDAP.  But in that sync configuration you are telling the provider to use the value returned by the "cn" attribute as the user identity.  This I suspect would return just "Myuser".  So NiFi would treat these two identities as different users. Now looking at your ldap-provider (user for user authentication) and see this concern: <property name="User Search Filter">(sAMAccountName={0})</property>  This ldap-provider performs a ldap search where the username entered at the NiFi login window is substituted in place of the "{0}".   Are you using AD (commonly where I'd expect to see a "sAMAccountName" attribute to be used)?   If this attribute does not exist on your ldap/AD users, your login is not going to be successful.  Perhaps you meant to use "cn={0}" here? I also see that your have configured the following in the ldap-provider:  <property name="Identity Strategy">USE_USERNAME</property>  This property tells the ldap-provider what to use as the user's identity post successful authentication. With "USE_USERNAME", the username (case sensitive) enter in the login window is used.  "USE_DN" will use the full DN returned by ldap/AD as the user identity. So with your current setup i expect your authorization will fail as your "username" will be passed to your configured authorizer where in your file-access-policy provider you have configured your initial admin to use a full DN. Additionally, NIFi provides configuration properties that can be used to manipulate the user identity and or group identity strings returned via authentication should you choose to go that route. https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#identity-mapping-properties So once you get past your login window issue, you'll need to resolve above. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@JobBranwl    A secured NiFi will always "WANT" a clientAuth certificate for client/user authentication even if you have configured additional method of user authentication. So when you access the NiFi UI your browser returns the clientAuth certificate you loaded in the TLS exchange.  When No other method of user/client authentication is configured, NiFi will "REQUIRE" a userAuth certificate.  When alternate method of authentication are configured, NiFi will "WANT" a clientAuth certificate.  This cannot be disabled.  NiFi nodes need to communicate with one another in a cluster and the Nodes clientAuth certificates are used to facilitate that authentication and authorization. Other features of NiFi like Site-To-Site also will uses a mutualTLS auth. Removing the client certificate from your browser or using a different browser where you have not loaded that certificate would result in NiFi moving on to next configured authentication method (provided those alternate methods are properly configured). Also important that you understand that modifications to the file-user-group-provider and/or file-access-policy provider will not result in those changes being applied to existing users.xml and/or authorizations.xml files those providers generate.    Based on your exception for your certificate based nifiadmin user, I am guessing you did remove these files so they got regenerated on startup using the newly defined initial admin Identity. When you say you changed your nifi.properties file, i assume that to mean you changed the configuration for the login provider: nifi.security.user.login.identity.provider=ldap-provider Try using a different browser where you have not installed the nifiadmin certificate. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@J_123  This is a 6 year old post.  NiFi Templates are close to point of deprecation from NiFi.  A NiFi is and has always been a snapshot of a specific set of processors.  It has no mechanism for tracking its reuse within the same NiFi where it was used.  Lets say someone creates a template, reuses it on the same NiFi multiple times, and then deletes the template from the NiFi (which is recommended since template like the rest of the flow on he canvas reside in NIFi's heap memory), how then do we track and replicate those changes.  Memory usage and this tracking are a few reasons why templates are in the way out and why we no longer recommend using this capability. Back at the end of 2021 (long after this post), NiFi introduced a new NiFi-Registry[1] service that is installed separate form NiFi and integrates with the NiFi service.  NiFi-Registry allows users to build dataflows on the a NiFi canvas and then version control each of those flows into NiFi-Registry.  Once version controlled into NiFi-Registry, that version controlled flow can be reuses on the same NiFi or any other NiFi using that same NiFi-Registry. If a user makes a change to anyone of these version flows linked to version control, NiFi will prompt that local changes exist allowing user to commit those locL changes to the copy in the NiFi-Registry.  Then all the other places (across multiple NiFi's if using that same dataflow from NiFi-Registry) where that version controlled flow is being used will prompt that a newer version is available allowing users to apply those changes.   If you have questions about something in NiFi and you cannot find a recent thread on the topic to raise a new question in the community to get the most up-to-date information. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt   [1] https://nifi.apache.org/docs/nifi-registry-docs/index.html ... View more

@Girish007  The stack trace above shows this exception of concern: Caused by: org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2022-12-14T11:18:50Z, (lock=NativeFSLock(path=/flowfile_repo/nifi/nifi-1.16.2/provenance_repository/lucene-8-index-1671016730507/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2022-12-14T11:18:50.521826Z)) 1. You appear to placed your provenance_repository inside of the same directory as your flowfile_repository?   2. The exception stated that the "Underlying file changed by an external force".  This indicates that Some service or script outside of NiFi is modifying these files.  Typically when we see this it is caused by some virus scanning service.  You should make sure that NiFi's repositories are excluded form scanning by any virus software.  This would explain why it works until it does not. 3. Since some external interaction with the provenance repository has introduced corruption there, you will need to stop NiFi, Exclude the provenance_repository (any other repositories while at it), delete the current contents of provenance_repository since it is corrupt, and then start NiFi again. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more