question Re: NiFi 1.15 cluster node error ConnectionLoadBalanceServer Failed to communicate with Peer node in Support Questions

NiFi 1.15 cluster node error ConnectionLoadBalanceServer Failed to communicate with Peer node

ZhouJun — Wed, 08 Dec 2021 12:10:22 GMT

Hi all, We have a NiFi 1.15 cluster with five nodes, after running for a while there is a node with below error, I couldn't find the root reason, please help me, thanks.

2021-12-08 19:58:35,803 ERROR [Load-Balance Server Thread-4385] o.a.n.c.q.c.s.ConnectionLoadBalanceServer Failed to communicate with Peer p-edi-nifi-003.gz.cvte.cn/10.21.46.32:55354 org.apache.nifi.controller.queue.clustered.server.TransactionAbortedException: Attempted to receive FlowFiles from Peer p-edi-nifi-003 for Connection with ID bf3245df-e6d7-15a9-0000-00005fd59914 but no Connection exists with that ID at org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFiles(StandardLoadBalanceProtocol.java:205) at org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFiles(StandardLoadBalanceProtocol.java:134) at org.apache.nifi.controller.queue.clustered.server.ConnectionLoadBalanceServer$CommunicateAction.run(ConnectionLoadBalanceServer.java:162) at java.base/java.lang.Thread.run(Thread.java:834) 2021-12-08 19:58:36,225 ERROR [Load-Balance Server Thread-4386] o.a.n.c.q.c.s.StandardLoadBalanceProtocol Attempted to receive FlowFiles from Peer p-edi-nifi-002 for Connection with ID cd623c09-a674-1975-b364-57fe9ffea60a but no connection exists with that ID

The cluster status are normal.

nifi.properties about load balance

# cluster load balancing properties # nifi.cluster.load.balance.host=p-edi-nifi-001 nifi.cluster.load.balance.port=6342 nifi.cluster.load.balance.connections.per.node=5 nifi.cluster.load.balance.max.thread.count=8 nifi.cluster.load.balance.comms.timeout=30 sec

Re: NiFi 1.15 cluster node error ConnectionLoadBalanceServer Failed to communicate with Peer node

ZhouJun — Wed, 15 Dec 2021 01:49:55 GMT

now all nodes have the same issue and after running a day , some nodes will disconnect from cluster, I upload node 2 and node 3's dump file and nifi-app.log, who can help me locate the cause, thanks in advance.

the primary and coordinate node is p-edi-nifi-003:9443

node3 nifi-app.log:

2021-12-15 09:47:37,851 ERROR [Load-Balanced Client Thread-2] o.a.n.c.queue.SwappablePriorityQueue Updated Size of Queue Unacknowledged from FlowFile Queue Size[ ActiveQueue=[1, 2424 Bytes], Swap Queue=[0, 0 Bytes], Swap Files=[0], Unacknowledged=[-1, -2424 Bytes] ] to FlowFile Queue Size[ ActiveQueue=[1, 2424 Bytes], Swap Queue=[0, 0 Bytes], Swap Files=[0], Unacknowledged=[-1, -2424 Bytes] ] java.lang.RuntimeException: Cannot create negative queue size at org.apache.nifi.controller.queue.SwappablePriorityQueue.logIfNegative(SwappablePriorityQueue.java:1055) at org.apache.nifi.controller.queue.SwappablePriorityQueue.incrementUnacknowledgedQueueSize(SwappablePriorityQueue.java:1045) at org.apache.nifi.controller.queue.SwappablePriorityQueue.acknowledge(SwappablePriorityQueue.java:451) at org.apache.nifi.controller.queue.clustered.partition.RemoteQueuePartition$1.onTransactionFailed(RemoteQueuePartition.java:172) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:271) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:81) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) 2021-12-15 09:47:37,893 ERROR [Load-Balanced Client Thread-3] o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient Failed to communicate with Peer p-edi-nifi-002:9443 java.io.EOFException: Encountered End-of-File when trying to verify with Peer p-edi-nifi-002:9443 whether or not space is available in Connection 124d3a0f-0921-13e8-ffff-ffffd6530ec7 at org.apache.nifi.controller.queue.clustered.client.async.nio.LoadBalanceSession.receiveSpaceAvailableResponse(LoadBalanceSession.java:584) at org.apache.nifi.controller.queue.clustered.client.async.nio.LoadBalanceSession.communicate(LoadBalanceSession.java:151) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:264) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:81) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) 2021-12-15 09:47:37,893 ERROR [Load-Balanced Client Thread-3] o.a.n.c.queue.SwappablePriorityQueue Updated Size of Queue Unacknowledged from FlowFile Queue Size[ ActiveQueue=[1, 2424 Bytes], Swap Queue=[0, 0 Bytes], Swap Files=[0], Unacknowledged=[-1, -2424 Bytes] ] to FlowFile Queue Size[ ActiveQueue=[1, 2424 Bytes], Swap Queue=[0, 0 Bytes], Swap Files=[0], Unacknowledged=[-1, -2424 Bytes] ] java.lang.RuntimeException: Cannot create negative queue size at org.apache.nifi.controller.queue.SwappablePriorityQueue.logIfNegative(SwappablePriorityQueue.java:1055) at org.apache.nifi.controller.queue.SwappablePriorityQueue.incrementUnacknowledgedQueueSize(SwappablePriorityQueue.java:1045) at org.apache.nifi.controller.queue.SwappablePriorityQueue.acknowledge(SwappablePriorityQueue.java:451) at org.apache.nifi.controller.queue.clustered.partition.RemoteQueuePartition$1.onTransactionFailed(RemoteQueuePartition.java:172) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:271) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:81) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) 2021-12-15 09:47:38,153 ERROR [Load-Balanced Client Thread-10] o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient Failed to communicate with Peer p-edi-nifi-002:9443 java.io.EOFException: Encountered End-of-File when trying to verify with Peer p-edi-nifi-002:9443 whether or not space is available in Connection 124d3a0f-0921-13e8-ffff-ffffd6530ec7 at org.apache.nifi.controller.queue.clustered.client.async.nio.LoadBalanceSession.receiveSpaceAvailableResponse(LoadBalanceSession.java:584) at org.apache.nifi.controller.queue.clustered.client.async.nio.LoadBalanceSession.communicate(LoadBalanceSession.java:151) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:264) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:81) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) 2021-12-15 09:47:38,153 ERROR [Load-Balanced Client Thread-10] o.a.n.c.queue.SwappablePriorityQueue Updated Size of Queue Unacknowledged from FlowFile Queue Size[ ActiveQueue=[1, 2424 Bytes], Swap Queue=[0, 0 Bytes], Swap Files=[0], Unacknowledged=[-1, -2424 Bytes] ] to FlowFile Queue Size[ ActiveQueue=[1, 2424 Bytes], Swap Queue=[0, 0 Bytes], Swap Files=[0], Unacknowledged=[-1, -2424 Bytes] ] java.lang.RuntimeException: Cannot create negative queue size at org.apache.nifi.controller.queue.SwappablePriorityQueue.logIfNegative(SwappablePriorityQueue.java:1055) at org.apache.nifi.controller.queue.SwappablePriorityQueue.incrementUnacknowledgedQueueSize(SwappablePriorityQueue.java:1045) at org.apache.nifi.controller.queue.SwappablePriorityQueue.acknowledge(SwappablePriorityQueue.java:451) at org.apache.nifi.controller.queue.clustered.partition.RemoteQueuePartition$1.onTransactionFailed(RemoteQueuePartition.java:172) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:271) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:81) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) 2021-12-15 09:47:38,154 ERROR [Load-Balanced Client Thread-11] o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient Failed to communicate with Peer p-edi-nifi-002:9443 java.io.EOFException: Encountered End-of-File when trying to verify with Peer p-edi-nifi-002:9443 whether or not space is available in Connection 124d3a0f-0921-13e8-ffff-ffffd6530ec7 at org.apache.nifi.controller.queue.clustered.client.async.nio.LoadBalanceSession.receiveSpaceAvailableResponse(LoadBalanceSession.java:584) at org.apache.nifi.controller.queue.clustered.client.async.nio.LoadBalanceSession.communicate(LoadBalanceSession.java:151) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:264) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:81) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) 2021-12-15 09:47:38,154 ERROR [Load-Balanced Client Thread-11] o.a.n.c.queue.SwappablePriorityQueue Updated Size of Queue Unacknowledged from FlowFile Queue Size[ ActiveQueue=[1, 2424 Bytes], Swap Queue=[0, 0 Bytes], Swap Files=[0], Unacknowledged=[-1, -2424 Bytes] ] to FlowFile Queue Size[ ActiveQueue=[1, 2424 Bytes], Swap Queue=[0, 0 Bytes], Swap Files=[0], Unacknowledged=[-1, -2424 Bytes] ] java.lang.RuntimeException: Cannot create negative queue size at org.apache.nifi.controller.queue.SwappablePriorityQueue.logIfNegative(SwappablePriorityQueue.java:1055) at org.apache.nifi.controller.queue.SwappablePriorityQueue.incrementUnacknowledgedQueueSize(SwappablePriorityQueue.java:1045) at org.apache.nifi.controller.queue.SwappablePriorityQueue.acknowledge(SwappablePriorityQueue.java:451) at org.apache.nifi.controller.queue.clustered.partition.RemoteQueuePartition$1.onTransactionFailed(RemoteQueuePartition.java:172) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:271) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:81) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) 2021-12-15 09:47:38,154 ERROR [Load-Balanced Client Thread-8] o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient Failed to communicate with Peer p-edi-nifi-002:9443 java.io.EOFException: Encountered End-of-File when trying to verify with Peer p-edi-nifi-002:9443 whether or not space is available in Connection 124d3a0f-0921-13e8-ffff-ffffd6530ec7 at org.apache.nifi.controller.queue.clustered.client.async.nio.LoadBalanceSession.receiveSpaceAvailableResponse(LoadBalanceSession.java:584) at org.apache.nifi.controller.queue.clustered.client.async.nio.LoadBalanceSession.communicate(LoadBalanceSession.java:151) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient.communicate(NioAsyncLoadBalanceClient.java:264) at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:81) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)

node3 dump log:

"Timer-Driven Process Thread-235" Id=20239 WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@2ed0512e at java.base@11.0.12/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.12/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.12/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885) at java.base@11.0.12/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(AbstractQueuedSynchronizer.java:943) at java.base@11.0.12/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1263) at java.base@11.0.12/java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:317) at java.base@11.0.12/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1165) at java.base@11.0.12/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11.0.12/java.lang.Thread.run(Thread.java:834) "Timer-Driven Process Thread-236" Id=20240 BLOCKED on org.apache.nifi.controller.scheduling.LifecycleState@7797e692 at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:148) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base@11.0.12/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base@11.0.12/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base@11.0.12/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11.0.12/java.lang.Thread.run(Thread.java:834) Number of Locked Synchronizers: 1 - java.util.concurrent.ThreadPoolExecutor$Worker@27a30d22 "Timer-Driven Process Thread-237" Id=20241 BLOCKED on org.apache.nifi.controller.scheduling.LifecycleState@19ddb845 at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:148) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base@11.0.12/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base@11.0.12/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base@11.0.12/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11.0.12/java.lang.Thread.run(Thread.java:834) Number of Locked Synchronizers: 1 - java.util.concurrent.ThreadPoolExecutor$Worker@3546839b "Timer-Driven Process Thread-238" Id=20242 BLOCKED on org.apache.nifi.controller.scheduling.LifecycleState@5be285e1 at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:148) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base@11.0.12/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base@11.0.12/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base@11.0.12/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11.0.12/java.lang.Thread.run(Thread.java:834) Number of Locked Synchronizers: 1 - java.util.concurrent.ThreadPoolExecutor$Worker@7feca0fd "Timer-Driven Process Thread-239" Id=20243 BLOCKED on org.apache.nifi.controller.scheduling.LifecycleState@50c8d0aa at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:148) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base@11.0.12/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base@11.0.12/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base@11.0.12/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11.0.12/java.lang.Thread.run(Thread.java:834) Number of Locked Synchronizers: 1 - java.util.concurrent.ThreadPoolExecutor$Worker@346f2d55 "Timer-Driven Process Thread-24" Id=125 WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@2ed0512e at java.base@11.0.12/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.12/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.12/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885) at java.base@11.0.12/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireInterruptibly(AbstractQueuedSynchronizer.java:943) at java.base@11.0.12/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1263) at java.base@11.0.12/java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:317) at java.base@11.0.12/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1165) at java.base@11.0.12/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11.0.12/java.lang.Thread.run(Thread.java:834) "Timer-Driven Process Thread-240" Id=20244 BLOCKED on org.apache.nifi.controller.scheduling.LifecycleState@326199cd at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:148) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base@11.0.12/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base@11.0.12/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base@11.0.12/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11.0.12/java.lang.Thread.run(Thread.java:834) Number of Locked Synchronizers: 1 - java.util.concurrent.ThreadPoolExecutor$Worker@2de59cd9 "Timer-Driven Process Thread-241" Id=20245 WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@2ed0512e at java.base@11.0.12/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.12/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.12/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885) at java.base@11.0.12/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:917) at java.base@11.0.12/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1240) at java.base@11.0.12/java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:267) at java.base@11.0.12/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.offer(ScheduledThreadPoolExecutor.java:1100) at java.base@11.0.12/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.add(ScheduledThreadPoolExecutor.java:1127) at java.base@11.0.12/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.add(ScheduledThreadPoolExecutor.java:899) at java.base@11.0.12/java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:342) at java.base@11.0.12/java.util.concurrent.ScheduledThreadPoolExecutor.scheduleWithFixedDelay(ScheduledThreadPoolExecutor.java:680) at org.apache.nifi.engine.FlowEngine.scheduleWithFixedDelay(FlowEngine.java:97) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:150) - waiting on org.apache.nifi.controller.scheduling.LifecycleState@19ddb845 at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base@11.0.12/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base@11.0.12/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base@11.0.12/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11.0.12/java.lang.Thread.run(Thread.java:834) Number of Locked Synchronizers: 1 - java.util.concurrent.ThreadPoolExecutor$Worker@7cfd3756 "Timer-Driven Process Thread-242" Id=20246 BLOCKED on org.apache.nifi.controller.scheduling.LifecycleState@656f3849 at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:148) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base@11.0.12/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base@11.0.12/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base@11.0.12/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11.0.12/java.lang.Thread.run(Thread.java:834) Number of Locked Synchronizers: 1 - java.util.concurrent.ThreadPoolExecutor$Worker@ec7bba7

node2 nifi-app.log:

2021-12-15 09:14:59,071 ERROR [Load-Balance Server Thread-14224] o.a.n.c.q.c.s.ConnectionLoadBalanceServer Failed to communicate with Peer p-edi-nifi-003.gz.cvte.cn/10.21.46.32:43266 org.apache.nifi.controller.queue.clustered.server.TransactionAbortedException: Attempted to receive FlowFiles from Peer p-edi-nifi-003 for Connection with ID 124d3a0f-0921-13e8-ffff-ffffd6530ec7 but no Connection exists with that ID at org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFiles(StandardLoadBalanceProtocol.java:205) at org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFiles(StandardLoadBalanceProtocol.java:134) at org.apache.nifi.controller.queue.clustered.server.ConnectionLoadBalanceServer$CommunicateAction.run(ConnectionLoadBalanceServer.java:162) at java.base/java.lang.Thread.run(Thread.java:834) 2021-12-15 09:14:59,073 ERROR [Load-Balance Server Thread-14220] o.a.n.c.q.c.s.StandardLoadBalanceProtocol Attempted to receive FlowFiles from Peer p-edi-nifi-003 for Connection with ID 124d3a0f-0921-13e8-ffff-ffffd6530ec7 but no connection exists with that ID 2021-12-15 09:14:59,073 ERROR [Load-Balance Server Thread-14221] o.a.n.c.q.c.s.StandardLoadBalanceProtocol Attempted to receive FlowFiles from Peer p-edi-nifi-003 for Connection with ID 124d3a0f-0921-13e8-ffff-ffffd6530ec7 but no connection exists with that ID 2021-12-15 09:14:59,073 ERROR [Load-Balance Server Thread-14220] o.a.n.c.q.c.s.ConnectionLoadBalanceServer Failed to communicate with Peer p-edi-nifi-003.gz.cvte.cn/10.21.46.32:43256 org.apache.nifi.controller.queue.clustered.server.TransactionAbortedException: Attempted to receive FlowFiles from Peer p-edi-nifi-003 for Connection with ID 124d3a0f-0921-13e8-ffff-ffffd6530ec7 but no Connection exists with that ID at org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFiles(StandardLoadBalanceProtocol.java:205) at org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFiles(StandardLoadBalanceProtocol.java:134) at org.apache.nifi.controller.queue.clustered.server.ConnectionLoadBalanceServer$CommunicateAction.run(ConnectionLoadBalanceServer.java:162) at java.base/java.lang.Thread.run(Thread.java:834) 2021-12-15 09:14:59,073 ERROR [Load-Balance Server Thread-14221] o.a.n.c.q.c.s.ConnectionLoadBalanceServer Failed to communicate with Peer p-edi-nifi-003.gz.cvte.cn/10.21.46.32:43262 org.apache.nifi.controller.queue.clustered.server.TransactionAbortedException: Attempted to receive FlowFiles from Peer p-edi-nifi-003 for Connection with ID 124d3a0f-0921-13e8-ffff-ffffd6530ec7 but no Connection exists with that ID at org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFiles(StandardLoadBalanceProtocol.java:205) at org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFiles(StandardLoadBalanceProtocol.java:134) at org.apache.nifi.controller.queue.clustered.server.ConnectionLoadBalanceServer$CommunicateAction.run(ConnectionLoadBalanceServer.java:162) at java.base/java.lang.Thread.run(Thread.java:834) 2021-12-15 09:14:59,073 ERROR [Load-Balance Server Thread-14226] o.a.n.c.q.c.s.StandardLoadBalanceProtocol Attempted to receive FlowFiles from Peer p-edi-nifi-005 for Connection with ID b54a33c5-2264-1ded-ffff-ffffbd054c96 but no connection exists with that ID 2021-12-15 09:14:59,074 ERROR [Load-Balance Server Thread-14226] o.a.n.c.q.c.s.ConnectionLoadBalanceServer Failed to communicate with Peer p-edi-nifi-005.gz.cvte.cn/10.21.46.34:60262 org.apache.nifi.controller.queue.clustered.server.TransactionAbortedException: Attempted to receive FlowFiles from Peer p-edi-nifi-005 for Connection with ID b54a33c5-2264-1ded-ffff-ffffbd054c96 but no Connection exists with that ID at org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFiles(StandardLoadBalanceProtocol.java:205) at org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFiles(StandardLoadBalanceProtocol.java:134) at org.apache.nifi.controller.queue.clustered.server.ConnectionLoadBalanceServer$CommunicateAction.run(ConnectionLoadBalanceServer.java:162) at java.base/java.lang.Thread.run(Thread.java:834) 2021-12-15 09:14:59,076 ERROR [Load-Balance Server Thread-14218] o.a.n.c.q.c.s.StandardLoadBalanceProtocol Attempted to receive FlowFiles from Peer p-edi-nifi-003 for Connection with ID 124d3a0f-0921-13e8-ffff-ffffd6530ec7 but no connection exists with that ID 2021-12-15 09:14:59,076 ERROR [Load-Balance Server Thread-14218] o.a.n.c.q.c.s.ConnectionLoadBalanceServer Failed to communicate with Peer p-edi-nifi-003.gz.cvte.cn/10.21.46.32:43255 org.apache.nifi.controller.queue.clustered.server.TransactionAbortedException: Attempted to receive FlowFiles from Peer p-edi-nifi-003 for Connection with ID 124d3a0f-0921-13e8-ffff-ffffd6530ec7 but no Connection exists with that ID at org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFiles(StandardLoadBalanceProtocol.java:205) at org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFiles(StandardLoadBalanceProtocol.java:134) at org.apache.nifi.controller.queue.clustered.server.ConnectionLoadBalanceServer$CommunicateAction.run(ConnectionLoadBalanceServer.java:162) at java.base/java.lang.Thread.run(Thread.java:834) 2021-12-15 09:14:59,077 ERROR [Load-Balance Server Thread-14223] o.a.n.c.q.c.s.StandardLoadBalanceProtocol Attempted to receive FlowFiles from Peer p-edi-nifi-003 for Connection with ID 124d3a0f-0921-13e8-ffff-ffffd6530ec7 but no connection exists with that ID 2021-12-15 09:14:59,078 ERROR [Load-Balance Server Thread-14223] o.a.n.c.q.c.s.ConnectionLoadBalanceServer Failed to communicate with Peer p-edi-nifi-003.gz.cvte.cn/10.21.46.32:43264 org.apache.nifi.controller.queue.clustered.server.TransactionAbortedException: Attempted to receive FlowFiles from Peer p-edi-nifi-003 for Connection with ID 124d3a0f-0921-13e8-ffff-ffffd6530ec7 but no Connection exists with that ID at org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFiles(StandardLoadBalanceProtocol.java:205) at org.apache.nifi.controller.queue.clustered.server.StandardLoadBalanceProtocol.receiveFlowFiles(StandardLoadBalanceProtocol.java:134) at org.apache.nifi.controller.queue.clustered.server.ConnectionLoadBalanceServer$CommunicateAction.run(ConnectionLoadBalanceServer.java:162) at java.base/java.lang.Thread.run(Thread.java:834)

node2 dump file:

"StandardHttpContextMap-aca62685-0179-1000-ffff-ffff821b01ae" Id=204 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@57c01834 at java.base@11.0.12/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.12/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11.0.12/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.base@11.0.12/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11.0.12/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11.0.12/java.lang.Thread.run(Thread.java:834) "StandardHttpContextMap-dd9c9135-0178-1000-0000-0000464c518b" Id=153 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f1251b8 at java.base@11.0.12/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.12/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11.0.12/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.base@11.0.12/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11.0.12/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11.0.12/java.lang.Thread.run(Thread.java:834) "SyncThread:2" Id=88 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7e205cc7 at java.base@11.0.12/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.12/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11.0.12/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) at java.base@11.0.12/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433) at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:109) "ThreadPoolRequestReplicator Maintenance Thread" Id=122 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@138779d3 at java.base@11.0.12/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11.0.12/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11.0.12/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.base@11.0.12/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11.0.12/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11.0.12/java.lang.Thread.run(Thread.java:834) "Timer-1" Id=20698 WAITING on java.util.TaskQueue@779365a3 at java.base@11.0.12/java.lang.Object.wait(Native Method) at java.base@11.0.12/java.lang.Object.wait(Object.java:328) at java.base@11.0.12/java.util.TimerThread.mainLoop(Timer.java:527) at java.base@11.0.12/java.util.TimerThread.run(Timer.java:506) "Timer-Driven Process Thread-1" Id=76 BLOCKED on org.apache.nifi.controller.scheduling.LifecycleState@12189489 at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:148) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base@11.0.12/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base@11.0.12/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base@11.0.12/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11.0.12/java.lang.Thread.run(Thread.java:834) Number of Locked Synchronizers: 1 - java.util.concurrent.ThreadPoolExecutor$Worker@64236be "Timer-Driven Process Thread-10" Id=156 BLOCKED on org.apache.nifi.controller.scheduling.LifecycleState@3c217039 at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:148) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base@11.0.12/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base@11.0.12/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base@11.0.12/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11.0.12/java.lang.Thread.run(Thread.java:834) Number of Locked Synchronizers: 1 - java.util.concurrent.ThreadPoolExecutor$Worker@ca31114 "Timer-Driven Process Thread-100" Id=5721 BLOCKED on org.apache.nifi.controller.scheduling.LifecycleState@70dce42a at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:148) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base@11.0.12/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base@11.0.12/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base@11.0.12/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base@11.0.12/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11.0.12/java.lang.Thread.run(Thread.java:834) Number of Locked Synchronizers: 1 - java.util.concurrent.ThreadPoolExecutor$Worker@fdaafba

Re: NiFi 1.15 cluster node error ConnectionLoadBalanceServer Failed to communicate with Peer node

dupuy_gregory — Mon, 20 Dec 2021 17:28:33 GMT

Hi ZhouJun,

Have you find a solution to your issue ?

I have the problem in my production and staging environnement since few months without any solutions. Network issue maybe ?

Thanks for your feedback

Re: NiFi 1.15 cluster node error ConnectionLoadBalanceServer Failed to communicate with Peer node

Venkataphani — Tue, 08 Feb 2022 09:18:50 GMT

Hi,

have you found any solution for this? appreciate if you have any inputs on this regard.

Thanks

Re: NiFi 1.15 cluster node error ConnectionLoadBalanceServer Failed to communicate with Peer node

lhoss_Ansiblize — Tue, 02 Aug 2022 12:53:52 GMT

The issue above, specifically also containing error:

java.lang.RuntimeException: Cannot create negative queue size

seems to have been solved in Nifi v15.1 or v16!

See [NIFI-9433] Load Balanced Connections hangs / log "Cannot create negative queue size" - ASF JIRA (apache.org)

Myself, I have an issue in the same high-level error "ConnectionLoadBalanceServer Failed to communicate with Peer node" , but a different (root)cause ( which I'm not sure yet how it is caused , but must be related to network, our docker-swarm multi-node deployment OR a resource bottleneck):

... ERROR [Load-Balanced Client Thread-7] o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClient Failed to communicate with Peer nifi2:8443 java.net.SocketTimeoutException: Timed out waiting for Peer nifi2:8443

Re: NiFi 1.15 cluster node error ConnectionLoadBalanceServer Failed to communicate with Peer node

MattWho — Tue, 02 Aug 2022 13:17:58 GMT

@ZhouJun

I'd recommend upgrading your NiFi to the latest release as you may be hitting these related bugs:
https://issues.apache.org/jira/browse/NIFI-9835
https://issues.apache.org/jira/browse/NIFI-9433
https://issues.apache.org/jira/browse/NIFI-9761

Thank you,

Matt