Frequent Node Disconnects and Flow Synchronization Issues in NiFi 1.25 with3 node cluster , node-0 gets issue all the time

[#1]

I have nifi cluster hosted on AKS , its 3 node  with memory 20G each node. node has dedicated pvcs standard lrs.

We are seeing occasionally once in 1/2 weeks nifi node-0 only disconnects  with below error.

We are running full load ingestions frequently. but there is no pattern . we  run 3-4 processor groups  have overlapping time (parallel).  but not all the time we get error . most of the runs on most of the day work . sometime we get below mentioned error.   what we observed . node-0 first gets disconnected later when it tried to reconnect  org.apache.nifi.controller.serialization.FlowSynchronizationException.

Any help possible ?

Error dump : 

2025-11-01 06:05:19,513 WARN [Notify Cluster of Node Status Change-2] o.a.n.c.p.i.StandardClusterCoordinationProtocolSender Failed to send Node Status Change message to nifi-0.nifi-headless.nifi-highresource-lts.svc.cluster.local:8443

org.apache.nifi.cluster.protocol.ProtocolException: Failed to create socket due to: java.net.UnknownHostException: nifi-0.nifi-headless.nifi-highresource-lts.svc.cluster.local

at org.apache.nifi.cluster.protocol.impl.StandardClusterCoordinationProtocolSender.createSocket(StandardClusterCoordinationProtocolSender.java:216)

at org.apache.nifi.cluster.protocol.impl.StandardClusterCoordinationProtocolSender.createSocket(StandardClusterCoordinationProtocolSender.java:204)

at org.apache.nifi.cluster.protocol.impl.StandardClusterCoordinationProtocolSender.access$000(StandardClusterCoordinationProtocolSender.java:67)

at org.apache.nifi.cluster.protocol.impl.StandardClusterCoordinationProtocolSender$2.run(StandardClusterCoordinationProtocolSender.java:296)

at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

at java.base/java.util.concurrent.FutureTask.run(Unknown Source)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.base/java.lang.Thread.run(Unknown Source)

Caused by: java.net.UnknownHostException: nifi-0.nifi-headless.nifi-highresource-lts.svc.cluster.local

at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)

at java.base/java.net.SocksSocketImpl.connect(Unknown Source)

at java.base/java.net.Socket.connect(Unknown Source)

at java.base/sun.security.ssl.SSLSocketImpl.connect(Unknown Source)

at java.base/sun.security.ssl.SSLSocketImpl.<init>(Unknown Source)

at java.base/sun.security.ssl.SSLSocketFactoryImpl.createSocket(Unknown Source)

at org.apache.nifi.io.socket.SocketUtils.createSocket(SocketUtils.java:68)

at org.apache.nifi.cluster.protocol.impl.StandardClusterCoordinationProtocolSender.createSocket(StandardClusterCoordinationProtocolSender.java:210)

... 8 common frames omitted

2025-11-01 06:05:18,438 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 2 heartbeats in 20659 nanos

2025-11-01 06:05:18,463 INFO [NiFi Web Server-20322] o.a.n.c.c.node.NodeClusterCoordinator Event Reported for nifi-0.nifi-headless.nifi-highresource-lts.svc.cluster.local:8443 -- Requesting that node connect to cluster on behalf of username

2025-11-01 06:05:18,464 INFO [NiFi Web Server-20322] o.a.n.c.c.node.NodeClusterCoordinator Status of nifi-0.nifi-headless.nifi-highresource-lts.svc.cluster.local:8443 changed from NodeConnectionStatus[nodeId=nifi-0.nifi-headless.nifi-highresource-lts.svc.cluster.local:8443, state=DISCONNECTED, Disconnect Code=Node's Flow did not Match Cluster Flow, Disconnect Reason=org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed to connect node to cluster because local flow controller partially updated. Administrator should disconnect node and review flow for corruption., updateId=44] to NodeConnectionStatus[nodeId=nifi-0.nifi-headless.nifi-highresource-lts.svc.cluster.local:8443, state=CONNECTING, updateId=5]

2025-11-01 06:05:18,492 WARN [Notify Cluster of Node Status Change-2] o.a.n.c.p.i.StandardClusterCoordinationProtocolSender Failed to send Node Status Change message to nifi-0.nifi-headless.nifi-highresource-lts.svc.cluster.local:8443

org.apache.nifi.cluster.protocol.ProtocolException: Failed to create socket due to: java.net.UnknownHostException: nifi-0.nifi-headless.nifi-highresource-lts.svc.cluster.local

[#2]

@Sanga 

What reason is being logged by the elected cluster coordinator node when node-0 gets initially disconnected?  Is it a lack of heartbeat?

I understand that onc it gets initially disconnected, NiFi cluster coordinator requests that it reconnect on next heartbeat and the triggered flow synchronization result in a exception blocking the node from rejoining the cluster.

My initial thought is  that you may be hitting NIFI-12969 or NIFI-12232 which was addressed in Apache NiFi 1.26.  An upgrade may help with your issue.

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt