I have nifi cluster hosted on AKS , its 3 node with memory 20G each node. node has dedicated pvcs standard lrs.
We are seeing occasionally once in 1/2 weeks nifi node-0 only disconnects with below error.
We are running full load ingestions frequently. but there is no pattern . we run 3-4 processor groups have overlapping time (parallel). but not all the time we get error . most of the runs on most of the day work . sometime we get below mentioned error. what we observed . node-0 first gets disconnected later when it tried to reconnect org.apache.nifi.controller.serialization.FlowSynchronizationException.
Any help possible ?
Error dump :
2025-11-01 06:05:19,513 WARN [Notify Cluster of Node Status Change-2] o.a.n.c.p.i.StandardClusterCoordinationProtocolSender Failed to send Node Status Change message to nifi-0.nifi-headless.nifi-highresource-lts.svc.cluster.local:8443
org.apache.nifi.cluster.protocol.ProtocolException: Failed to create socket due to: java.net.UnknownHostException: nifi-0.nifi-headless.nifi-highresource-lts.svc.cluster.local
at org.apache.nifi.cluster.protocol.impl.StandardClusterCoordinationProtocolSender.createSocket(StandardClusterCoordinationProtocolSender.java:216)
at org.apache.nifi.cluster.protocol.impl.StandardClusterCoordinationProtocolSender.createSocket(StandardClusterCoordinationProtocolSender.java:204)
at org.apache.nifi.cluster.protocol.impl.StandardClusterCoordinationProtocolSender.access$000(StandardClusterCoordinationProtocolSender.java:67)
at org.apache.nifi.cluster.protocol.impl.StandardClusterCoordinationProtocolSender$2.run(StandardClusterCoordinationProtocolSender.java:296)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.net.UnknownHostException: nifi-0.nifi-headless.nifi-highresource-lts.svc.cluster.local
at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.base/java.net.SocksSocketImpl.connect(Unknown Source)
at java.base/java.net.Socket.connect(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.connect(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.<init>(Unknown Source)
at java.base/sun.security.ssl.SSLSocketFactoryImpl.createSocket(Unknown Source)
at org.apache.nifi.io.socket.SocketUtils.createSocket(SocketUtils.java:68)
at org.apache.nifi.cluster.protocol.impl.StandardClusterCoordinationProtocolSender.createSocket(StandardClusterCoordinationProtocolSender.java:210)
... 8 common frames omitted
2025-11-01 06:05:18,438 INFO [Heartbeat Monitor Thread-1] o.a.n.c.c.h.AbstractHeartbeatMonitor Finished processing 2 heartbeats in 20659 nanos
2025-11-01 06:05:18,463 INFO [NiFi Web Server-20322] o.a.n.c.c.node.NodeClusterCoordinator Event Reported for nifi-0.nifi-headless.nifi-highresource-lts.svc.cluster.local:8443 -- Requesting that node connect to cluster on behalf of username
2025-11-01 06:05:18,464 INFO [NiFi Web Server-20322] o.a.n.c.c.node.NodeClusterCoordinator Status of nifi-0.nifi-headless.nifi-highresource-lts.svc.cluster.local:8443 changed from NodeConnectionStatus[nodeId=nifi-0.nifi-headless.nifi-highresource-lts.svc.cluster.local:8443, state=DISCONNECTED, Disconnect Code=Node's Flow did not Match Cluster Flow, Disconnect Reason=org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed to connect node to cluster because local flow controller partially updated. Administrator should disconnect node and review flow for corruption., updateId=44] to NodeConnectionStatus[nodeId=nifi-0.nifi-headless.nifi-highresource-lts.svc.cluster.local:8443, state=CONNECTING, updateId=5]
2025-11-01 06:05:18,492 WARN [Notify Cluster of Node Status Change-2] o.a.n.c.p.i.StandardClusterCoordinationProtocolSender Failed to send Node Status Change message to nifi-0.nifi-headless.nifi-highresource-lts.svc.cluster.local:8443
org.apache.nifi.cluster.protocol.ProtocolException: Failed to create socket due to: java.net.UnknownHostException: nifi-0.nifi-headless.nifi-highresource-lts.svc.cluster.local
