@davehkd  When your nodes become disconnected, a reason will be logged and also most recent events viewable from within the cluster UI via the NIFi interface.   So first question is reason given for node disconnections?  Is it reporting a communication exception with Zookeeper or is it reporting disconnection due to lack of heartbeat (more common). Within a cluster a node is elected as the cluster coordinator by ZK, the nodes begin sending health and status heartbeats to that cluster coordinator. Default is every 5 seconds.  The elected cluster coordinator expects to receive at least one heartbeat every 8x the configured heartbeat interval, so every 40 seconds.  This is a pretty aggressive setting for NiFi clusters under heavy load or high heap pressure caused by dataflow design.  So first make sure that every node in your cluster has the same configured heartbeat interval value (mixed values will definitely cause lots of node disconnections).  If you are seeing reason for disconnection as lack of heartbeat, adjust the heartbeat interval to 30 seconds.   This means a heartbeat would need to missed in a 4 minutes window instead of 40 seconds. As far as GC goes, GC is triggered when Java heap utilization gets around ~80%.  How much memory have you configured your NiFi to use?  Setting really high for no reason means would result in longer GC stop-the-world events.   Generally NiFi would be configured with 16 GB to 32 GB for most use cases. If you find yourself needing more then that , you should take a closer look at your dataflow implementations (dataflows).  The NiFi heap holds many things including the following: - fllow.json.gz is unpacked and loaded into heap memory on startup.  Flow.json.gz includes everything you have added and configured via the NiFi UI (flows, controller settings, registry clients, templates, etc.). Templates are a deprecated method of creating flow snippets for reuse.  They are held in heap because they are part of the flow.json.gz even though they are not part of any active dataflow.  Downloading for external storage and deleting from within NiFi will reduce heap usage. - user and groups synced from ldap if using the ldap-user-group-provider.  Shoudl make sure that your have configured filters on this provider so that you are liimiting the number of groups and users to only those the will actually be accessing yoru NiFi. - FlowFiles are what you see queued between processor components on the UI.  FlowFiles consist of metatdata/attributes about the FlowFile.  NiFi has build in swap settings for how many FlowFiles can exist in a given queue before they start swapping to disk (20,000 set via nifi.queue.swap.threshold in nifi.properties).  Swap files are always 10,000 FlowFiles. By default, a connection has a backpressure object threshold of 10,000.  This means by default a connection will not likely generate a swap file because it is unlikely to reach the swap threshold with these defaults (connection queues are soft limits).  So If you have lots of connection with queued FlowFiles, you will have more heap usage.  Generally speaking, a FlowFile's default metadata attributes amount to very little heap usage, but users can write whatever they want to FlowFile attributes. If you extracting and writing larges amounts of content to FlowFile attributes in yoru dataflow(s), you'll have high heap usage and should be question yourself as to why you are doing this. - NiFi processor components - Some processors have resource considerations that users should take in to considerations when using those processors.  The embedded documentation within your NiFi will have section for resource considerations under each processor's docs.  Look to see if you are using and with heap/memory consideration. Often heap usage can be reduced through dataflow design modifications. I hope these details help you dig into your heap usage and helps you make adjustments to improve your cluster stability. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt ... View more

@apmmahesh  You created certificates for each of your NiFi nodes.  Base on exception you shared, it appears that you created DNs for those nodes as following? CN=node1, OU=NIFI CN=node2, OU=NIFI CN=node3, OU=NIFI When you have a NiFi cluster, you can manage that cluster via the UI of any one of the connected nodes. So let's say you authenticate via a mutual TLS handshake to node1 using your CN=admin, OU=NIFI certificate you created for yourself and loaded in your browser.  What happens next is node1 wants to show you all the data/details from all three nodes and not just node1, so your request to load the NiFi is sent via proxy by node1 to whichever node is the elected cluster coordinator.  That cluster coordinator replicates the request on your behalf to all nodes in the cluster.  This how the node1 UI would show you details about connected nodes, queued data from other nodes, etc.  This means that node1 would need to be authorized to proxy user requests.  So typically on first startup secure NiFi will use the configuration in your authorizers.xml to setup these needed default authorization, but your configuration is missing your nodes, so this was not done. Inside your file user-group-provider, you need to also add your NiFi node DNs as users. <userGroupProvider> <identifier>file-user-group-provider</identifier> <class>org.apache.nifi.authorization.FileUserGroupProvider</class> <property name="Users File">./conf/users.xml</property> <property name="Legacy Authorized Users File"></property> <property name="Initial User Identity 1">CN=admin, OU=NIFI</property> <property name="Initial User Identity 2">CN=node1, OU=NIFI</property> <property name="Initial User Identity 3">CN=node2, OU=NIFI</property> <property name="Initial User Identity 4">CN=node3, OU=NIFI</property> </userGroupProvider> Then in your file-access-policy-provider you need to add your nodes so that when it generates the authorizations.xml file, the nodes get authorized to the "proxy user requests" policy: <accessPolicyProvider> <identifier>file-access-policy-provider</identifier> <class>org.apache.nifi.authorization.FileAccessPolicyProvider</class> <property name="User Group Provider">file-user-group-provider</property> <property name="Authorizations File">./conf/authorizations.xml</property> <property name="Initial Admin Identity">CN=admin, OU=NIFI</property> <property name="Legacy Authorized Users File"></property> <property name="Node Identity 1">CN=node1, OU=NIFI</property> <property name="Node Identity 2">CN=node2, OU=NIFI</property> <property name="Node Identity 3">CN=node3, OU=NIFI</property> <property name="Node Group"></property> </accessPolicyProvider> NOTE:  NiFI will only create the users.xml and authorizations.xml files from the above two providers if they do NOT already exist.  Making changes to these providers will not result in changes to existing files.  The expectation is that after access for yoru initial admin and your proxy nodes is established that all new authorizations are setup via the NiFi UI which will result in updated to these files.  So rename your existing users.xml and authorizations.xml before starting yoru NiFi so new get created. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt   ... View more