@hckorkmaz01  @vafs provided a couple excellent options already. As a third option, you could utilize the SiteToSiteStatusReportingTask.  You can configure this NiFi reporting task to send metrics for only connections and you can add a regex to limited the number of connections returned (this requires renaming the desired connections so you have more granular control with the regex).  This Reporting task can be configured to send to a remote input port on on the same NiFi.   This Reporting task will execute the query on each host in your NiFi Cluster and send results to the the desired Site-To-Site remote input port.   So in a three node NiFi cluster three FlowFiles will be produced and sent with Json content for connection(s) you filtered on.   This way you can see what is queued per node.  Limitations to monitoring connections in this way exist.  Keep in mind that you are grabbing a snapshot at one specific moment in time.  So if the downstream component of this monitored connection is running, the connection stat would have likely changed by the time you got the last status details.   Now if you are using this to monitor flow dead-ends where you don't expect FlowFiles to ever accumulate, that is perhaps a valid use case. Example json output placed in FlowFile: [ { "statusId" : "27b01368-8280-4e5e-ac22-0e749c46fe17", "timestampMillis" : 1771336855410, "timestamp" : "2026-02-17T14:00:55.410Z", "actorHostname" : "nifi-node-1", "componentType" : "Connection", "componentName" : "Monitored-success", "parentId" : "603017c1-0197-1000-0000-000013171e7c", "parentName" : "test", "parentPath" : "NiFi Flow / test", "platform" : "nifi", "application" : "NiFi Flow", "componentId" : "508a2293-019a-1000-ffff-fffff15dc582", "sourceId" : "5089d4ab-019a-1000-ffff-ffff9b682ab3", "sourceName" : "UpdateAttribute", "destinationId" : "4dc23b89-25b0-1eff-b974-6fe5425c283f", "destinationName" : "UpdateAttribute", "maxQueuedBytes" : 0, "maxQueuedCount" : 0, "queuedBytes" : 30720, "queuedCount" : 30, "inputBytes" : 0, "inputCount" : 0, "outputBytes" : 0, "outputCount" : 0, "backPressureBytesThreshold" : 1073741824, "backPressureObjectThreshold" : 10000, "backPressureDataSizeThreshold" : "1 GB", "isBackPressureEnabled" : "false" } ]   IMPORTANT NOTE: I'd also like to point out that Apache NiFi 1.18 was released back in 2021 and newer releases have addressed many bugs and CVEs.  I strongly encourage you to transition to a newer version of Apache NiFi.  If you aren't ready to migrate to the newer Apache NiFi 2.x major release versions, you can still easily upgrade to last Apache NiFi 1.x major release version (1.28.0). Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@zzzz77  Provenance can be very noisy depending on size of your dataflows and the amount of FlowFIles being processed through those dataflows.  The provenance repo has age and size configuration that trigger roll-off of old events.   So you may not reach the retention age if you reach size first.  Also would not be trying to read provenance files while they are being written to.    The SiteToSiteProvenanceReportingTask might be the solution you are looking for in Apache NiFi.   This reporting task will send all provenance events over Site-To-Site protocol to a target NiFi where you can then feed them into any long term storage medium of your choice in a human readable format.   Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@zzzz77   I can certainly help you with the structured setup commonly used when integrating NIFi with LDAP.  NiFi authentication and authorization are different processes and configurations.  You can even authenticate using LDAP and not use LDAP at all during authorization. Also need to be aware that only a secured NiFi setup over HTTPS can support authentication and authorization. Since Authentication needs to happen first, we'll start there. LDAP authentication is configured as a login provider inside the login-identity-providers.xml configuration file:   <provider> <identifier>ldap-provider</identifier> <class>org.apache.nifi.ldap.LdapProvider</class> <property name="Authentication Strategy">START_TLS</property> <property name="Manager DN"></property> <property name="Manager Password"></property> <property name="TLS - Keystore"></property> <property name="TLS - Keystore Password"></property> <property name="TLS - Keystore Type"></property> <property name="TLS - Truststore"></property> <property name="TLS - Truststore Password"></property> <property name="TLS - Truststore Type"></property> <property name="TLS - Client Auth"></property> <property name="TLS - Protocol"></property> <property name="TLS - Shutdown Gracefully"></property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url"></property> <property name="User Search Base"></property> <property name="User Search Filter"></property> <property name="Identity Strategy">USE_DN</property> <property name="Authentication Expiration">12 hours</property> </provider>   The actual configuration is dependent on your LDAP setup.  You can refer to the linked documentation for each field.  Depending on "Authentication Strategy" setting, TLS properties may not need to be configured.  The "identifier" for this provider is "ldap-provider".   The "Identity Strategy" is used to decide what string is used as the authenticated users identity.  Options are "USE_DN" (use the full DN from the LDAP entry) or "USE_USERNAME" (use the username as typed in the login window).   USE_USERNAME is commonly used. This identifier needs to be configured in the nifi.properties file, so NiFi knows which login-provider NiFi should be using.    nifi.security.user.login.identity.provider=ldap-provider     Now we need to setup the authorizers.xml file so we can setup authorizations for the ldap users.  Here you have two options, you can manually add the ldap user identities via the "user-group-provider" or you can sync the user identities directly from ldap using the "ldap-user-group-provider".  Sometimes you want both if not all your users/clients are part of LDAP (this applies to user identities derived from clientAuth certificates during a mutualTLS exchange).   Both would commonly be necessary for a NiFi cluster setup.   Since you are setting up a single instance (non cluster) NiFi, I'll show how to structure your authorizers.xml file using just the ldap-user-group-provider: <userGroupProvider> <identifier>ldap-user-group-provider</identifier> <class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class> <property name="Authentication Strategy">SIMPLE</property> <property name="Manager DN">cn=Manager,dc=nifi,dc=hwx</property> <property name="Manager Password">password</property> <property name="TLS - Keystore"></property> <property name="TLS - Keystore Password"></property> <property name="TLS - Keystore Type"></property> <property name="TLS - Truststore"></property> <property name="TLS - Truststore Password"></property> <property name="TLS - Truststore Type"></property> <property name="TLS - Client Auth"></property> <property name="TLS - Protocol"></property> <property name="TLS - Shutdown Gracefully"></property> <property name="Referral Strategy">FOLLOW</property> <property name="Connect Timeout">10 secs</property> <property name="Read Timeout">10 secs</property> <property name="Url">ldap://<ip or hostname>:389</property> <property name="Page Size">500</property> <property name="Sync Interval">30 mins</property> <property name="Group Membership - Enforce Case Sensitivity">false</property> <property name="User Search Base">ou=People,dc=nifi,dc=hwx</property> <property name="User Object Class">inetOrgPerson</property> <property name="User Search Scope">SUBTREE</property> <property name="User Search Filter"></property> <property name="User Identity Attribute">cn</property> <property name="User Group Name Attribute">memberOf</property> <property name="User Group Name Attribute - Referenced Group Attribute"></property> <property name="Group Search Base">ou=Group,dc=nifi,dc=hwx</property> <property name="Group Object Class">groupOfNames</property> <property name="Group Search Scope">SUBTREE</property> <property name="Group Search Filter"></property> <property name="Group Name Attribute">cn</property> <property name="Group Member Attribute">member</property> <property name="Group Member Attribute - Referenced User Attribute"></property> </userGroupProvider> <accessPolicyProvider> <identifier>file-access-policy-provider</identifier> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class> <property name="User Group Provider">ldap-user-group-provider</property> <property name="Authorizations File">./conf/authorizations.xml</property> <property name="Initial Admin Identity">nifiadmin</property> <property name="Node Identity 1"></property> <property name="Node Group"></property> </accessPolicyProvider> <authorizer> <identifier>managed-authorizer</identifier> <class>org.apache.nifi.authorization.StandardManagedAuthorizer</class> <property name="Access Policy Provider">file-access-policy-provider</property> </authorizer> Above authorizer is the most basic setup example assuming an unsecure ldap setup as the example. You can see it has three sections.  The bets way to read an authorizers.xml configuration is from the bottom up starting with the "authorizer".  In this example you can see I am using the "StandardManagedAuthorizer" which has an identifier of "managed-authorizer" and it is configured to reference the "file-access-policy-provider".  So the next provider we should find going up through the authorizers.xml will be the provider with the identifier "file-access-policy-provider". The "FileAccessPolicyProvider" is responsible for persisting the granted authorizations in a file name "authorizations.xml". This provider will also set some initial authorizations for the user identity set in the "Initial Admin Identity" field and the for any "Node Identity <num>" field entries.   We can see that this provider is learning about users and groups from the "ldap-user-group-provider". IMPORTANT NOTES:  This provider will only create the authorizations.xml file if it does NOT already exist.  So if you make any changes to this provider, those changes would not be reflected in an already existing authorizations.xml file.   Also any identity strings set this provider must be returned by a user-group-provider(s).  So the next provider needed has the identifier "ldap-user-group-provider" and needs to be located further up in this authorizations.xml file.  So we locate the "LdapUserGroupProvider" which has this identifier.  This provider has no reference to any additional providers.  While i shared a very basic sample configuration, your configuration will be specific to your ldap server source. My example is configured to sync users and groups from ldap.  You can choose to sync users or users and groups.  You can not sync just groups.   Inside the nifi.properties file you will set the authorizer you want to use: nifi.security.user.authorizer=managed-authorizer Now that we have the authentication and authorization setup complete, let's walk through what happens when you access NiFi's "https://<hostname>:<port>/nifi" url. A mutualTLS exchange with the client (browser) will occur where NiFi will "WANT" a clientAuth certificate.  Of one is not presented in that exchange, NiFi will redirect to the login UI: Here the user will supply their ldap username and password.   Assuming the ldap-login-identity-provider is using "USE_USERNAME" and authentication was successful, the username (case sensitive) as typed in the username field will be passed to the managed authorizer to check what authorizations are in place for that user.  Before that user identity reaches the managed authorizer, it is compared against the any Identity Mapping Properties configured in the nifi.properties file to see if any string manipulation should happen.  Next the string (manipulated if mapping was applied) goes to the authorizer.  First the authorizer will check to see if that user identity belongs to any groups.  Then it will check if the user or any groups that user is known to be member of (based on returns from ldap-user-group-provider sync) has proper authorizations to access the NiFi UI.   If proper authorization exist, you will see the NiFi UI and the user identity will show in the upper right corner. If there are authorization issues, you'll find that logged in the nifi-user.log.   Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@fy-test  I can't speak to your specific Zookeeper setup.  However, from a NiFi standpoint... NiFi-Registry has not dependency on Zookeeper, so it can be started at anytime. NiFi cluster setups have a requirement for zookeeper quorum before the NiFi cluster can be formed.  NiFi cluster can be started even without ZK quorum, but all nodes will be in a disconnected state until the ZK quorum is established and one of the NiFi cluster nodes is elected as the cluster coordinator by ZK, at which time all nodes will start sending heartbeats to that elected cluster coordinator and the cluster will be formed/established.  Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@Frank168  Unfortunately, Apache NiFi does not support Nested Groups. There is an existing Apache NiFi Jira (NIFI-8035) for such an improvement, but it has never been implemented.  The existing implementation of the ldap-user-group-provider would treat all members of a group as users and does not validate the type of member.  Any change here would require NiFi to retrieve the object class of all members of a group and then conduct another search of any that were of identified as a group to retrieve their members and so on until all users are identified throughout the entire nested group tree.    Something to keep in mind here is that all the user and group identities along with associations are held in the NiFi heap memory on every node.  So doing such could result in a lot of user and groups consuming NiFi heap memory.   You should configure your Ldap-user-group-provider to sync only the groups from which users exist that will be accessing your NiFi limiting the length of time it takes to sync every 30 minutes and the heap memory impact.   Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@fy-test  Starting only on ZK node will not give quorum, so the NiFi cluster would not form.   NiFi nodes would come up and continue to attempt to connect to ZK quorum for cluster coordinator election before cluster could form. All NiFi nodes need to learn which node is elected to this role in order to know which node to send heartbeats to in order to form a NiFi cluster.  I'd say you have some other issues if your ZK quorum cluster is not stable when NiFi is started.  My ZK is completely up with quorum when I start any of my NiFi clusters.  If Quorum keeps coming and going due to some issue in your ZK, that could cause NiFi nodes to disconnect from cluster and reconnect when quorum exists again.  The real question here is why is your ZK not coming up well when you start all of the ZK hosts at the same time.  I'd spend more time looking at the health of your ZK. All you should need to do is start ZK nodes so you have quorum and then start NiFi and NiFi-Registry (order of NiFi and NiFi-Registry start does not matter).   Thanks, Matt ... View more

@hegdemahendra  Did you take heap dumps to confirm which class was consuming the heap? Any thread dump analysis when heap usage was growing? What incorrect values were configured that you feel led to this component consuming large amounts of heap memory? Not really finding any known issues of memory with this consumeKafka processor.  Any details you can provide (processor configuration, log exceptions while it was running with bad config, etc) may help. Thank you, Matt ... View more

@fy-test  I would not expect this step to be necessary: Delete flow.json.gz on disconnected nodes → successful rejoin - Flow election happens during startup. Once a flow is elected, nodes that join afterwards will inherit the cluster flow if their local flow does not match.   I see no need to clear ZK state.   ZK elects a cluster coordinator and primary node from the nodes that establish connection with ZK.   ZK also used for components in your dataflows that utilize cluster state. Clearing ZK could result in duplicate data processing depending in what your flow does in NiFi. This is interesting line shared from your logs. no heartbeat from node in 15089 seconds This implies that the elected cluster coordinator disconnected a node after not receiving a heartbeat for 15089 seconds.  This means the node was in a connected state.  On startup of a cluster, all nodes are in a disconnected state initially until they connect, so this is not a line I would expect to see during startup.  Was this accompanied by a line that stated node was being disconnected due to lack of heartbeat? Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt ... View more

@fy-test  Apache NiFi 2.7.2 does not use a flow.xml.gz file (This format was only used by Apache NiFi 1.x versions).   Apache NiFi 2.x versions use a flow.json.gz format.  I would suggest making sure the Zookeeper quorum is up before starting the NiFi Service. NiFi cluster can't form or remain formed if Zookeeper does not have stable quorum. If your NiFi nodes are disconnecting and reconnecting, I would start by looking at the status of the nodes to see what reason is being given for the disconnects.   You can find this in the Cluster UI within NiFi: Clicking on the small "i" icon to the left of the node name will open the pop-up window above that shows node events.  You should also see node events in the nifi-app.logs on each node.  So you see your elected cluster coordinator constantly changing? Was there a duration of time between stop and start? Was there a large influx of backlogged data when NiFi was started? Encounter any OutOfMemory exceptions? Encounter any long garbage collection events? Did nifi-app.log on elected cluster coordinator(s) reported any node disconnected due to lack of heartbeat log output? You would normally start all nodes at the same time.  NiFi knows how many nodes were last in the cluster and has a flow election process that depends on all nodes connecting. So startup times will be mush longer if not all nodes connect.  NiFi has a configurable timer of how long flow election will run before finishing starting with just the nodes that connected.  Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt   ... View more