Support Questions

Find answers, ask questions, and share your expertise

one node is not starting in NIFI Cluster

Explorer

we have set up 4 node cluster and all other 3 nodes start without any exceptions. and in NIFI ui
it shows our down node is in connecting state.

but our down NIFI node is not running.

its logs as below.

2022-10-17 19:04:51,860 INFO [main] org.apache.nifi.io.socket.SocketListener Now listening for connections from nodes on port 11443
2022-10-17 19:04:51,950 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager Failed to close Leader Selector for Cluster Coordinator
java.lang.IllegalStateException: Already closed or has not been started
at com.google.common.base.Preconditions.checkState(Preconditions.java:173)
at org.apache.curator.framework.recipes.leader.LeaderSelector.close(LeaderSelector.java:270)
at org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager.stop(CuratorLeaderElectionManager.java:165)
at org.apache.nifi.controller.FlowController.shutdown(FlowController.java:1120)
at org.apache.nifi.controller.StandardFlowService.stop(StandardFlowService.java:349)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1016)
at org.apache.nifi.NiFi.<init>(NiFi.java:158)
at org.apache.nifi.NiFi.<init>(NiFi.java:72)
at org.apache.nifi.NiFi.main(NiFi.java:297)


java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClientTask.run(NioAsyncLoadBalanceClientTask.java:91)
at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2022-10-17 19:04:51,957 ERROR [Load-Balanced Client Thread-58] o.a.n.c.q.c.c.a.n.NioAsyncLoadBalanceClientTask Failed to communicate with peer while trying to load balance data across the cluster
java.lang.InterruptedException: sleep interrupted

2022-10-17 19:04:52,495 INFO [main] o.apache.nifi.controller.FlowController Controller has been terminated successfully.
2022-10-17 19:04:52,501 INFO [main] org.apache.nifi.io.socket.SocketListener Socket Listener has been terminated successfully.
2022-10-17 19:04:52,502 WARN [Cluster Socket Listener] org.apache.nifi.io.socket.SocketListener Failed to communicate with Unknown Host due to java.net.SocketException: Socket closed
java.net.SocketException: Socket closed
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at sun.security.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:348)
at org.apache.nifi.io.socket.SocketListener$2.run(SocketListener.java:112)
at java.lang.Thread.run(Thread.java:748)
2022-10-17 19:04:52,502 ERROR [main] org.apache.nifi.web.server.JettyServer Unable to load flow due to: java.util.zip.ZipException: invalid stored block lengths
java.util.zip.ZipException: invalid stored block lengths
at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:164)
at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:117)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at org.apache.nifi.util.file.FileUtils.copy(FileUtils.java:303)
at org.apache.nifi.persistence.StandardXMLFlowConfigurationDAO.load(StandardXMLFlowConfigurationDAO.java:110)
at org.apache.nifi.controller.StandardFlowService.createDataFlow(StandardFlowService.java:608)
at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:467)
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1009)
at org.apache.nifi.NiFi.<init>(NiFi.java:158)
at org.apache.nifi.NiFi.<init>(NiFi.java:72)
at org.apache.nifi.NiFi.main(NiFi.java:297)
2022-10-17 19:04:52,503 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.
java.lang.Exception: Unable to load flow due to: java.util.zip.ZipException: invalid stored block lengths
at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:1019)
at org.apache.nifi.NiFi.<init>(NiFi.java:158)
at org.apache.nifi.NiFi.<init>(NiFi.java:72)
at org.apache.nifi.NiFi.main(NiFi.java:297)
2022-10-17 19:04:52,504 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
2022-10-17 19:04:52,524 INFO [Thread-1] o.eclipse.jetty.server.AbstractConnector Stopped ServerConnector@4e3931{SSL,[ssl, http/1.1]}{bdadataf15.ncell.com.np:9443}
2022-10-17 19:04:52,525 INFO [Thread-1] org.eclipse.jetty.server.session node0 Stopped scavenging

we didn't do any changes in properties and we deployed one custom processor since its threw exceptions while processing we revert it back also.


1 ACCEPTED SOLUTION

Master Guru

@D5ha 

Sometimes it is useful to know more about your environment to include the full NiFi version and java versions.  

Since it is reporting issues as loading the flow:

java.lang.Exception: Unable to load flow due to: java.util.zip.ZipException: invalid stored block lengths
at org.apache.nifi.web.server.JettyServer.start

 I would lean towards some issue/corruption of the flow.xml.gz and/or flow.json.gz on this node.  Since all nodes run the same exact copy of these files, I'd copy them from a good node to the node failing to start.  Depending on your NiFi version you may not have a flow.json.gz file (This format was introduced in the most recent versions).


If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

View solution in original post

1 REPLY 1

Master Guru

@D5ha 

Sometimes it is useful to know more about your environment to include the full NiFi version and java versions.  

Since it is reporting issues as loading the flow:

java.lang.Exception: Unable to load flow due to: java.util.zip.ZipException: invalid stored block lengths
at org.apache.nifi.web.server.JettyServer.start

 I would lean towards some issue/corruption of the flow.xml.gz and/or flow.json.gz on this node.  Since all nodes run the same exact copy of these files, I'd copy them from a good node to the node failing to start.  Depending on your NiFi version you may not have a flow.json.gz file (This format was introduced in the most recent versions).


If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.