@phaelax  This is very possible and very commonly done. NiFi's Remote Process Group (RPG) is commonly used (recommended method) to transfer NiFi FlowFiles between different NiFi instances or even different NiFi clusters: https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Remote_Group_Transmission https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#remote_group_anatomy   When a direct connection between NiFi's can't be made, you can also use the mergeContent processor to "FlowFile Stream, v3" to merge numerous FlowFiles (includes FlowFile attributes) in to a single FlowFile that can be transferred by any means to another NiFi where it can be ingested and use the unpackContent processor to unplack that into the original FlowFiles with their original attributes.   If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@hegdemahendra  When updating a NiFi variable a serious of steps needs to occur. Steps To Update Variables Identifying components affected Stopping affected Processors Disabling affected Controller Services Applying Updates Re-Enabling affected Controller Services Restarting affected Processors So in this process what can lead to this taking a long time to complete is the Stopping of processors and disabling of controller services using that updated variable.  When NiFi requests a component to stop it transitions to a "stopping" or "disabling" stage.  During this phase the component will not linger be scheduled and the process waits for any existing threads being executed by those components to complete. Those threads do not get interrupted.  So when this take aa long time or "infinite" amount of time, troubleshooting this would require getting a series of thread dumps to see which threads are long running or perhaps hung preventing the impacted components from competing the thread execution that blocks the component from transition to a fully stopped or disabled state.  Understand that nothing in  a thread dump is going to directly point back to a very specific processor.  So it is important in your troubleshooting that you know what processors use the variable you are updating and look for threads that appear in the complete series of multiple thread dumps that relate back to those component classes.  Also in cases were you see these long running calls, are they for a variables used consistently by the same set of components to help narrow your analysis. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt    ... View more

@JobBranwl  So your ldap-user-group-provider is returning the value from the "cn" attribute which is likely just: initial-admin  However, you set your initial admin identity in the file-access-policy provider to a full DN.  So when that provider attempts to build the authorization.xml, it can't because neither provider returned that identity string. I recommend you leave Identity strategy as "USE_USERNAME" (most commonly used). I also recommend setting your initial admin identity in the file-access-policy provider to your admin username and not the full DN.  You can also add a logger to the NiFi logback.xml that will result in all the user and group identities strings being output in the nifi-app.log for inspection by you to aid in yoru troubleshooting here:  <logger name="org.apache.nifi.ldap.tenants.LdapUserGroupProvider" level="DEBUG"/>  Just scroll down in your logback until you start seeing logger lines and add this one. User and groups produced by the user-group providers are assigned a uuid.  That uuid is then used when assigning policies in the authorization.xml. That is why a user-group provider must return an identity in order to set permissions or seed initial policies against a user.  That is why you got that exception. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@ajignacio  In the stack trace yoou'll see below: Caused by: org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force   This indicates that something external to your NiFi is changing the provenance files.  When we see this it is most commonly the result of some virus software scanning the NiFi repositories and during that modifying the the files.    You should make sure that any external service exclude the NiFi repositories from scanning and modification. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@JobBranwl  Within your authorizers.xml file you can NOT have multiple user-group providers return the same user identity string. I see you creating the following user identity string in the file-user-group-provider   cn=Myuser,ou=Myuser,dc=Mydomain,dc=com    and also see in your ldap-user-group-provider that you may be trying to sync the sme user from your LDAP.  But in that sync configuration you are telling the provider to use the value returned by the "cn" attribute as the user identity.  This I suspect would return just "Myuser".  So NiFi would treat these two identities as different users. Now looking at your ldap-provider (user for user authentication) and see this concern: <property name="User Search Filter">(sAMAccountName={0})</property>  This ldap-provider performs a ldap search where the username entered at the NiFi login window is substituted in place of the "{0}".   Are you using AD (commonly where I'd expect to see a "sAMAccountName" attribute to be used)?   If this attribute does not exist on your ldap/AD users, your login is not going to be successful.  Perhaps you meant to use "cn={0}" here? I also see that your have configured the following in the ldap-provider:  <property name="Identity Strategy">USE_USERNAME</property>  This property tells the ldap-provider what to use as the user's identity post successful authentication. With "USE_USERNAME", the username (case sensitive) enter in the login window is used.  "USE_DN" will use the full DN returned by ldap/AD as the user identity. So with your current setup i expect your authorization will fail as your "username" will be passed to your configured authorizer where in your file-access-policy provider you have configured your initial admin to use a full DN. Additionally, NIFi provides configuration properties that can be used to manipulate the user identity and or group identity strings returned via authentication should you choose to go that route. https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#identity-mapping-properties So once you get past your login window issue, you'll need to resolve above. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@JobBranwl    A secured NiFi will always "WANT" a clientAuth certificate for client/user authentication even if you have configured additional method of user authentication. So when you access the NiFi UI your browser returns the clientAuth certificate you loaded in the TLS exchange.  When No other method of user/client authentication is configured, NiFi will "REQUIRE" a userAuth certificate.  When alternate method of authentication are configured, NiFi will "WANT" a clientAuth certificate.  This cannot be disabled.  NiFi nodes need to communicate with one another in a cluster and the Nodes clientAuth certificates are used to facilitate that authentication and authorization. Other features of NiFi like Site-To-Site also will uses a mutualTLS auth. Removing the client certificate from your browser or using a different browser where you have not loaded that certificate would result in NiFi moving on to next configured authentication method (provided those alternate methods are properly configured). Also important that you understand that modifications to the file-user-group-provider and/or file-access-policy provider will not result in those changes being applied to existing users.xml and/or authorizations.xml files those providers generate.    Based on your exception for your certificate based nifiadmin user, I am guessing you did remove these files so they got regenerated on startup using the newly defined initial admin Identity. When you say you changed your nifi.properties file, i assume that to mean you changed the configuration for the login provider: nifi.security.user.login.identity.provider=ldap-provider Try using a different browser where you have not installed the nifiadmin certificate. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@Girish007  The stack trace above shows this exception of concern: Caused by: org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2022-12-14T11:18:50Z, (lock=NativeFSLock(path=/flowfile_repo/nifi/nifi-1.16.2/provenance_repository/lucene-8-index-1671016730507/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2022-12-14T11:18:50.521826Z)) 1. You appear to placed your provenance_repository inside of the same directory as your flowfile_repository?   2. The exception stated that the "Underlying file changed by an external force".  This indicates that Some service or script outside of NiFi is modifying these files.  Typically when we see this it is caused by some virus scanning service.  You should make sure that NiFi's repositories are excluded form scanning by any virus software.  This would explain why it works until it does not. 3. Since some external interaction with the provenance repository has introduced corruption there, you will need to stop NiFi, Exclude the provenance_repository (any other repositories while at it), delete the current contents of provenance_repository since it is corrupt, and then start NiFi again. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@ajignacio  1. When you created the Provenance repository, did you move the old provenance data there or start new with an empty directory?    If the provenance repo is corrupt, you can start fresh by deleting everything in the provenance_repository directory.  Provenance data does not impact your active FlowFiles queued with your NiFi dataflows, so no data loss there.  All you lose is the lineage history that eventually ages off anyway. 2. What Java JDK version is your NiFi Using? 3. Can you share the complete stack trace from the nifi-app.log? 4. Any other errors/warns in the app.log (Memory or open file limit exceptions, any other warns or exceptions related to provenance, etc...)? Thank you, Matt ... View more