@Vishesh  Sorry, but there is not enough information to give a definitive cause for your issue.  There is typically full stack traces to go with these types of logged exceptions which can tell us more, but even then it may not be enough information still.   As i mentioned before there are numerous improvements between Apache NIFi 1.23 and the last Apache NIFi 1.x release version 1.28.  I'd strongly encourage you to upgrade to see if your issue persists.   You may be hitting this bug  NIFI-12232  which was addressed in versions 1.26+.  Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

Not so much resolved as worked around.  We had a large number of components on the canvas.  And we think we tracked the issue to timeouts in the cluster sync process.  (It was taking too long sometimes to get flow.xml sync'd between the nodes so it would stop.) The default timeout is 30 seconds.  We moved it up to 180 seconds. In Cloudera Manager, for the NiFi Configuration, search for properties which start with `nifi.cluster` to find the ones we changed. I also think things might have gotten better with more recent releases.  We have not seen this happen for quite some time. ... View more

@Frank168  I'd expect what you are seeing with the long sync times.  Sync interval has not impact in speed of the user an group syncs.  This simply controls how often NiFi is expected re-sync with ldap.  So in your case, NiFi would finding one sync and immediately start the next.  NiFi does a sync during startup as it loads the authorizers.xml, NiFi does not continue to load until that initial sync completes otherwise it can't do authorizations.   After that initial successful sync with ldap, NiFi will finish loading and only then will UI become available.   While running the sync interval will run in the background to re-sync from ldap in case any users or groups are added/removed/updated since last sync.    some users adjust this to sync every 24 hours instead of every 30 minutes.   This depends on how quickly you want NiFi to be aware of changes. As I commented before, you are loading 50,000+ users into NiFi heap memory (this impact the free heap available to the NiFi process for executing your dataflows on the canvas) and then redoing that every sync interval.  This was not how NiFi ldap-user-group-provider was intended to be used.   The only users and groups that should be loaded in to NiFi are those you plan to establish authorization policies for and will be accessing your secured NiFi. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Krish98  Unfortunately, I have no "Best Practice" recommendation for the use case you have shared.  It is not a use case I have ever setup before. Also want to share that in Apache NiFi 2.4+ version a new JWTBearerOAuth2AccessTokenProvider controller service was introduced.  While not a solution to you query, I wanted to share this with you.  Apache NiFi jira: NIFI-14380 NOTE: The Apache NiFi 1.x major release line is End-Of-Life now.  There will be no future releases of the 1. major release line.  There is no direct upgrade path from Apache NiFi 1.x to Apache NiFi 2.x.  You'll need to migrate your dataflows from 1.x to 2.x. For our Cloudera Flow Management licensed users, we provide tooling to assist with migrating dataflows from Flow Management versions based on Apache NiFi 1.x to Flow Management versions based on Apache NiFi 2.x.   Cloudera Flow Management 2.x also includes many of the components deprecated and no longer included in the Apache NiFi 2.x release line.  Thank you, Matt ... View more

Thank you @mat . ... View more

@hckorkmaz01  @vafs provided a couple excellent options already. As a third option, you could utilize the SiteToSiteStatusReportingTask.  You can configure this NiFi reporting task to send metrics for only connections and you can add a regex to limited the number of connections returned (this requires renaming the desired connections so you have more granular control with the regex).  This Reporting task can be configured to send to a remote input port on on the same NiFi.   This Reporting task will execute the query on each host in your NiFi Cluster and send results to the the desired Site-To-Site remote input port.   So in a three node NiFi cluster three FlowFiles will be produced and sent with Json content for connection(s) you filtered on.   This way you can see what is queued per node.  Limitations to monitoring connections in this way exist.  Keep in mind that you are grabbing a snapshot at one specific moment in time.  So if the downstream component of this monitored connection is running, the connection stat would have likely changed by the time you got the last status details.   Now if you are using this to monitor flow dead-ends where you don't expect FlowFiles to ever accumulate, that is perhaps a valid use case. Example json output placed in FlowFile: [ { "statusId" : "27b01368-8280-4e5e-ac22-0e749c46fe17", "timestampMillis" : 1771336855410, "timestamp" : "2026-02-17T14:00:55.410Z", "actorHostname" : "nifi-node-1", "componentType" : "Connection", "componentName" : "Monitored-success", "parentId" : "603017c1-0197-1000-0000-000013171e7c", "parentName" : "test", "parentPath" : "NiFi Flow / test", "platform" : "nifi", "application" : "NiFi Flow", "componentId" : "508a2293-019a-1000-ffff-fffff15dc582", "sourceId" : "5089d4ab-019a-1000-ffff-ffff9b682ab3", "sourceName" : "UpdateAttribute", "destinationId" : "4dc23b89-25b0-1eff-b974-6fe5425c283f", "destinationName" : "UpdateAttribute", "maxQueuedBytes" : 0, "maxQueuedCount" : 0, "queuedBytes" : 30720, "queuedCount" : 30, "inputBytes" : 0, "inputCount" : 0, "outputBytes" : 0, "outputCount" : 0, "backPressureBytesThreshold" : 1073741824, "backPressureObjectThreshold" : 10000, "backPressureDataSizeThreshold" : "1 GB", "isBackPressureEnabled" : "false" } ]   IMPORTANT NOTE: I'd also like to point out that Apache NiFi 1.18 was released back in 2021 and newer releases have addressed many bugs and CVEs.  I strongly encourage you to transition to a newer version of Apache NiFi.  If you aren't ready to migrate to the newer Apache NiFi 2.x major release versions, you can still easily upgrade to last Apache NiFi 1.x major release version (1.28.0). Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

Thanks @MattWho  Probably as a result of a nifikop-operator based, LDAP-Authenticated, Clustered Nifi setup, I had to give permissions to both the DN you mentioned and my nifi user (as visible on the top right corner of my nifi UI). Effective Permissions: - Can Proxy User Requests and Can Manage Buckets to both the Users - Read, Write and Delete (if required) access to my buckets Now I'm able to list and interact with my buckets from my nifi UI! ... View more

@zzzz77  Provenance can be very noisy depending on size of your dataflows and the amount of FlowFIles being processed through those dataflows.  The provenance repo has age and size configuration that trigger roll-off of old events.   So you may not reach the retention age if you reach size first.  Also would not be trying to read provenance files while they are being written to.    The SiteToSiteProvenanceReportingTask might be the solution you are looking for in Apache NiFi.   This reporting task will send all provenance events over Site-To-Site protocol to a target NiFi where you can then feed them into any long term storage medium of your choice in a human readable format.   Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@zzzz77   I can certainly help you with the structured setup commonly used when integrating NIFi with LDAP.  NiFi authentication and authorization are different processes and configurations.  You can even authenticate using LDAP and not use LDAP at all during authorization. Also need to be aware that only a secured NiFi setup over HTTPS can support authentication and authorization. Since Authentication needs to happen first, we'll start there. LDAP authentication is configured as a login provider inside the login-identity-providers.xml configuration file:   <provider> <identifier>ldap-provider</identifier> <class>org.apache.nifi.ldap.LdapProvider</class> <property name="Authentication Strategy">START_TLS</property> <property name="Manager DN"></property> <property name="Manager Password"></property> <property name="TLS - Keystore"></property> <property name="TLS - Keystore Password"></property> <property name="TLS - Keystore Type"></property> <property name="TLS - Truststore"></property> <property name="TLS - Truststore Password"></property> <property name="TLS - Truststore Type"></property> <property name="TLS - Client Auth"></property> <property name="TLS - Protocol"></property> <property name="TLS - Shutdown Gracefully"></property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url"></property> <property name="User Search Base"></property> <property name="User Search Filter"></property> <property name="Identity Strategy">USE_DN</property> <property name="Authentication Expiration">12 hours</property> </provider>   The actual configuration is dependent on your LDAP setup.  You can refer to the linked documentation for each field.  Depending on "Authentication Strategy" setting, TLS properties may not need to be configured.  The "identifier" for this provider is "ldap-provider".   The "Identity Strategy" is used to decide what string is used as the authenticated users identity.  Options are "USE_DN" (use the full DN from the LDAP entry) or "USE_USERNAME" (use the username as typed in the login window).   USE_USERNAME is commonly used. This identifier needs to be configured in the nifi.properties file, so NiFi knows which login-provider NiFi should be using.    nifi.security.user.login.identity.provider=ldap-provider     Now we need to setup the authorizers.xml file so we can setup authorizations for the ldap users.  Here you have two options, you can manually add the ldap user identities via the "user-group-provider" or you can sync the user identities directly from ldap using the "ldap-user-group-provider".  Sometimes you want both if not all your users/clients are part of LDAP (this applies to user identities derived from clientAuth certificates during a mutualTLS exchange).   Both would commonly be necessary for a NiFi cluster setup.   Since you are setting up a single instance (non cluster) NiFi, I'll show how to structure your authorizers.xml file using just the ldap-user-group-provider: <userGroupProvider> <identifier>ldap-user-group-provider</identifier> <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class> <property name="Authentication Strategy">SIMPLE</property> <property name="Manager DN">cn=Manager,dc=nifi,dc=hwx</property> <property name="Manager Password">password</property> <property name="TLS - Keystore"></property> <property name="TLS - Keystore Password"></property> <property name="TLS - Keystore Type"></property> <property name="TLS - Truststore"></property> <property name="TLS - Truststore Password"></property> <property name="TLS - Truststore Type"></property> <property name="TLS - Client Auth"></property> <property name="TLS - Protocol"></property> <property name="TLS - Shutdown Gracefully"></property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url">ldap://<ip or hostname>:389</property> <property name="Page Size">500</property> <property name="Sync Interval">30 mins</property> <property name="Group Membership - Enforce Case Sensitivity">false</property> <property name="User Search Base">ou=People,dc=nifi,dc=hwx</property> <property name="User Object Class">inetOrgPerson</property> <property name="User Search Scope">SUBTREE</property> <property name="User Search Filter"></property> <property name="User Identity Attribute">cn</property> <property name="User Group Name Attribute">memberOf</property> <property name="User Group Name Attribute - Referenced Group Attribute"></property> <property name="Group Search Base">ou=Group,dc=nifi,dc=hwx</property> <property name="Group Object Class">groupOfNames</property> <property name="Group Search Scope">SUBTREE</property> <property name="Group Search Filter"></property> <property name="Group Name Attribute">cn</property> <property name="Group Member Attribute">member</property> <property name="Group Member Attribute - Referenced User Attribute"></property> </userGroupProvider> <accessPolicyProvider> <identifier>file-access-policy-provider</identifier> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class> <property name="User Group Provider">ldap-user-group-provider</property> <property name="Authorizations File">./conf/authorizations.xml</property> <property name="Initial Admin Identity">nifiadmin</property> <property name="Node Identity 1"></property> <property name="Node Group"></property> </accessPolicyProvider> <authorizer> <identifier>managed-authorizer</identifier> <class>org.apache.nifi.authorization.StandardManagedAuthorizer</class> <property name="Access Policy Provider">file-access-policy-provider</property> </authorizer> Above authorizer is the most basic setup example assuming an unsecure ldap setup as the example. You can see it has three sections.  The bets way to read an authorizers.xml configuration is from the bottom up starting with the "authorizer".  In this example you can see I am using the "StandardManagedAuthorizer" which has an identifier of "managed-authorizer" and it is configured to reference the "file-access-policy-provider".  So the next provider we should find going up through the authorizers.xml will be the provider with the identifier "file-access-policy-provider". The "FileAccessPolicyProvider" is responsible for persisting the granted authorizations in a file name "authorizations.xml". This provider will also set some initial authorizations for the user identity set in the "Initial Admin Identity" field and the for any "Node Identity <num>" field entries.   We can see that this provider is learning about users and groups from the "ldap-user-group-provider". IMPORTANT NOTES:  This provider will only create the authorizations.xml file if it does NOT already exist.  So if you make any changes to this provider, those changes would not be reflected in an already existing authorizations.xml file.   Also any identity strings set this provider must be returned by a user-group-provider(s).  So the next provider needed has the identifier "ldap-user-group-provider" and needs to be located further up in this authorizations.xml file.  So we locate the "LdapUserGroupProvider" which has this identifier.  This provider has no reference to any additional providers.  While i shared a very basic sample configuration, your configuration will be specific to your ldap server source. My example is configured to sync users and groups from ldap.  You can choose to sync users or users and groups.  You can not sync just groups.   Inside the nifi.properties file you will set the authorizer you want to use: nifi.security.user.authorizer=managed-authorizer Now that we have the authentication and authorization setup complete, let's walk through what happens when you access NiFi's "https://<hostname>:<port>/nifi" url. A mutualTLS exchange with the client (browser) will occur where NiFi will "WANT" a clientAuth certificate.  Of one is not presented in that exchange, NiFi will redirect to the login UI: Here the user will supply their ldap username and password.   Assuming the ldap-login-identity-provider is using "USE_USERNAME" and authentication was successful, the username (case sensitive) as typed in the username field will be passed to the managed authorizer to check what authorizations are in place for that user.  Before that user identity reaches the managed authorizer, it is compared against the any Identity Mapping Properties configured in the nifi.properties file to see if any string manipulation should happen.  Next the string (manipulated if mapping was applied) goes to the authorizer.  First the authorizer will check to see if that user identity belongs to any groups.  Then it will check if the user or any groups that user is known to be member of (based on returns from ldap-user-group-provider sync) has proper authorizations to access the NiFi UI.   If proper authorization exist, you will see the NiFi UI and the user identity will show in the upper right corner. If there are authorization issues, you'll find that logged in the nifi-user.log.   Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@fy-test  I can't speak to your specific Zookeeper setup.  However, from a NiFi standpoint... NiFi-Registry has not dependency on Zookeeper, so it can be started at anytime. NiFi cluster setups have a requirement for zookeeper quorum before the NiFi cluster can be formed.  NiFi cluster can be started even without ZK quorum, but all nodes will be in a disconnected state until the ZK quorum is established and one of the NiFi cluster nodes is elected as the cluster coordinator by ZK, at which time all nodes will start sending heartbeats to that elected cluster coordinator and the cluster will be formed/established.  Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more