Failed to connect node to cluster because local flow controller partially updated. Administrator should disconnect node and review flow for corruption

[#1]

Hi, 

Nifi keep disconnecting due to below error , appreciated any help .

Node disconnected from cluster due to org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed to connect node to cluster because local flow controller partially updated. Administrator should disconnect node and review flow for corruption.
2023-09-21 12:34:09,572 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager Cannot unregister Leader Election Role 'Primary Node' because that role is not registered

[#2]

@Sivagopal 

What version of Apache NiFi are you using?
There are some bugs that could lead to this. I recommend you are using Apache NiFi version 1.21 or newer.

Is it always the same node that is disconnecting or appear random?
Was the ERROR in the nifi-app.log followed by a full stack trace?

- Make sure all nodes in your NiFi cluster are running with the same version of Apache NiFi.
- Make sure that any custom or add-on nars and jars where added to all every node and have proper ownership and permissions.
- Make sure all nodes use the same configured sensitive.props key and algorithm.

Depending on your version of Apache NiFi, you may have a flow.xml.gz or both a flow.xml.gz and flow.json.gz files.  On the node that is disconnecting, stop the node, and then rename the existing flow.xml.gz and flow.json.gz (if exists).  If you restart NiFi, it will inherit the flow from the cluster when it joins and write new flow.xml.gz and flow.json.gz files to disk.

If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#3]

We are having this problem now with CFM 2.1.5.0 (NiFi 1.18) and have no later version at present we can upgrade to.

It is not always the same node...but it may always be the primary node.  (At least it was the two times it happened today.)

Are there particular usage patterns which you know that trigger this problem with node synchronization that we can try to avoid?

Thanks.

[#4]

@kellerj 
CFM has several Service pack versions released for 2.1.5, as well as newer CFM 2.1.6 and CM 2.1.7 versions.

If you open the cluster UI via the NiFi UI --> global menu upper right corner) and then click on the "View Details" icon to far left of node that is disconnecting, what Node Events are being reported?

Matt

[#5]

It's really just the same message as the original.  (Un)fortunately, 2 of the 3 nodes of the cluster just dropped a few min ago so:

7/11/2024 16:57:12 UTC: Node disconnected due to org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed to connect node to cluster because local flow controller partially updated. Administrator should disconnect node and review flow for corruption.
07/11/2024 16:57:12 UTC: Disconnection requested due to org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed to connect node to cluster because local flow controller partially updated. Administrator should disconnect node and review flow for corruption.
07/11/2024 16:57:07 UTC: Node Status changed from CONNECTED to DISCONNECTED due to org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed to connect node to cluster because local flow controller partially updated. Administrator should disconnect node and review flow for corruption.
07/11/2024 16:56:56 UTC: Received first heartbeat from connecting node. Node connected.
07/11/2024 16:56:54 UTC: Requesting that node connect to cluster