@Siva227
The error you shared is when the node is trying to reconnect to the cluster following a disconnection. So first we need to identify why the node disconnected originally. I suspect you are disconnecting due to lack of heartbeat or your node failed to process a change request from the cluster coordinator node.
- Cluster Size: 6 nodes
- Each Node: 32 vCPUs, 256 GB RAM
- JVM Heap Memory: 192 GB (configured per node)
- Max Timer Driven Thread Count: 192
- Processor Count: Over 10,000 processors across the flows
Any specific reason why you configured your NiFi to use so much heap memory. Large heaps like this result in long stop-the-world Garbage Collections (GC). These long garbage collection stop-the -world events can lead to disconnections as a result of lack of heartbeat from that node. A common mistake is setting heap very large simply becuase you have a lot of memory on the node. You want to use the smallest heap possible as needed by yoru dataflows. GC does not kick in until heap usage reaches ~80%.
The below property controls heartbeat interval and lack of heartbeat disconnection:
nifi.cluster.protocol.heartbeat.interval=5 sec
The cluster coordinator will disconnect a node due lack of heartbeat if a heartbeat has not been received for 8 times this configured value (40 seconds in this case). It is very possible you encounter GC that last longer then this. I recommend changing your heartbeat interval to 30 sec which will allow up 4 mins of missed heartbeats before the cluster coordinator will disconnect a node.
The following error shared, while not initial cause of node disconnection, is preventing node from reconnecting:
Node disconnected due to Proposed Authorizer is not inheritable by the Flow Controller because NiFi has already started the dataflow and Authorizer has differences: Proposed Authorizations do not match current Authorizations: Proposed fingerprint is not inheritable because the current access policies is not empty.
This implies that there are differences in the authorizations.xml file on this node versus what the cluster has in its authorizations.xml. You also state this is the error seen ver often after a node disconnection?
Are you often modifying or setting up new authorization access policy when you have a node disconnect?
I'd start with identifying the initial cause of node disconnection which I suspect is either lack of heartbeat or failed to replicate request to node resulting in node being disconnected. Both of which can happen with long GC pauses.
Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.
Thank you,
Matt
