one node is not starting in NIFI Cluster

D5ha — Mon, 17 Oct 2022 15:37:30 GMT

we have set up 4 node cluster and all other 3 nodes start without any exceptions. and in NIFI ui
it shows our down node is in connecting state.

but our down NIFI node is not running.

its logs as below.

2022-10-17 19:04:51,860 INFO [main] org.apache.nifi.io.socket.SocketListener Now listening for connections from nodes on port 11443
2022-10-17 19:04:51,950 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager Failed to close Leader Selector for Cluster Coordinator
java.lang.IllegalStateException: Already closed or has not been started
at com.google.common.base.Preconditions.checkState(Preconditions.java:173)
at org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270)
at org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:165)
at org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1120)
at org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:349)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1016)
at org.apache.nifi.NiFi.<init>(NiFi.java:158)
at org.apache.nifi.NiFi.<init>(NiFi.java:72)
at org.apache.nifi.NiFi.main(NiFi.java:297)

java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:91)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2022-10-17 19:04:51,957 ERROR [Load-Balanced Client Thread-58] o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClientTask Failed to communicate with peer while trying to load balance data across the cluster
java.lang.InterruptedException: sleep interrupted

2022-10-17 19:04:52,495 INFO [main] o.apache.nifi.controller.FlowController Controller has been terminated successfully.
2022-10-17 19:04:52,501 INFO [main] org.apache.nifi.io.socket.SocketListener Socket Listener has been terminated successfully.
2022-10-17 19:04:52,502 WARN [Cluster Socket Listener] org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown Host due to java.net.SocketException: Socket closed
java.net.SocketException: Socket closed
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
at org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112)
at java.lang.Thread.run(Thread.java:748)
2022-10-17 19:04:52,502 ERROR [main] org.apache.nifi.web.server.JettyServer Unable to load flow due to: java.util.zip.ZipException: invalid stored block lengths
java.util.zip.ZipException: invalid stored block lengths
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:164)
at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:117)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at org.apache.nifi.util.file.FileUtils.copy(FileUtils.java:303)
at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:110)
at org.apache.nifi.controller.StandardFlowService.createDataFlow(StandardFlowService.java:608)
at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:467)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1009)
at org.apache.nifi.NiFi.<init>(NiFi.java:158)
at org.apache.nifi.NiFi.<init>(NiFi.java:72)
at org.apache.nifi.NiFi.main(NiFi.java:297)
2022-10-17 19:04:52,503 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.
java.lang.Exception: Unable to load flow due to: java.util.zip.ZipException: invalid stored block lengths
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1019)
at org.apache.nifi.NiFi.<init>(NiFi.java:158)
at org.apache.nifi.NiFi.<init>(NiFi.java:72)
at org.apache.nifi.NiFi.main(NiFi.java:297)
2022-10-17 19:04:52,504 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
2022-10-17 19:04:52,524 INFO [Thread-1] o.eclipse.jetty.server.AbstractConnector Stopped ServerConnector@4e3931{SSL,[ssl, http/1.1]}{bdadataf15.ncell.com.np:9443}
2022-10-17 19:04:52,525 INFO [Thread-1] org.eclipse.jetty.server.session node0 Stopped scavenging

we didn't do any changes in properties and we deployed one custom processor since its threw exceptions while processing we revert it back also.

Re: one node is not starting in NIFI Cluster

MattWho — Mon, 24 Oct 2022 16:13:57 GMT

@D5ha

Sometimes it is useful to know more about your environment to include the full NiFi version and java versions.

Since it is reporting issues as loading the flow:

java.lang.Exception: Unable to load flow due to: java.util.zip.ZipException: invalid stored block lengths at org.apache.nifi.web.server.JettyServer.start

I would lean towards some issue/corruption of the flow.xml.gz and/or flow.json.gz on this node. Since all nodes run the same exact copy of these files, I'd copy them from a good node to the node failing to start. Depending on your NiFi version you may not have a flow.json.gz file (This format was introduced in the most recent versions).

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

question one node is not starting in NIFI Cluster in Support Questions

one node is not starting in NIFI Cluster

Re: one node is not starting in NIFI Cluster