Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NiFi nodes restart randomly, no errors

NiFi nodes restart randomly, no errors

I have a 3 node Apache NiFi setup and I noticed that randomly Nifi on nodes restart. I do not see any errors in the logs. Any help is appreciated.

Logs right before restart

2018-03-29 17:07:19,364 INFO [Process Cluster Protocol Request-5] o.a.n.c.p.impl.SocketProtocolListener Finished processing request f0454ef3-7033-4510-b9f9-7fb1f7682c82 (type=NODE_STATUS_CHANGE, length=983 bytes) from 10.192.1.3 in 3 millis
2018-03-29 17:07:19,365 INFO [Leader Election Notification Thread-1] o.apache.nifi.controller.FlowController This node has been elected Primary Node
2018-03-29 17:07:19,367 INFO [Leader Election Notification Thread-2] o.a.n.c.l.e.CuratorLeaderElectionManager org.apache.nifi.controller.leader.election.CuratorLeaderElectionManager$ElectionListener@4521daa6 This node has been elected Leader for Role 'Cluster Coordinator'
2018-03-29 17:07:19,367 INFO [Leader Election Notification Thread-2] o.apache.nifi.controller.FlowController This node elected Active Cluster Coordinator
2018-03-29 17:07:23,470 INFO [Process Cluster Protocol Request-6] o.a.n.c.p.impl.SocketProtocolListener Finished processing request a4cebca8-26cc-4219-bf90-6ba24e4f0e41 (type=HEARTBEAT, length=3194 bytes) from 10.192.1.1:8080 in 3 millis
2018-03-29 17:07:24,168 INFO [Process Cluster Protocol Request-7] o.a.n.c.p.impl.SocketProtocolListener Finished processing request b01d2c79-418e-41a4-a432-7d54e456595a (type=HEARTBEAT, length=3196 bytes) from 10.192.1.2:8080 in 1 millis
2018-03-29 17:07:24,168 INFO [Clustering Tasks Thread-3] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-03-29 17:07:24,163 and sent to 10.192.1.2:18090 at 2018-03-29 17:07:24,168; send took 5 millis
2018-03-29 17:07:25,029 INFO [pool-2-thread-2] org.apache.nifi.BootstrapListener Received SHUTDOWN request from Bootstrap
2018-03-29 17:07:25,030 INFO [pool-2-thread-2] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
2018-03-29 17:07:25,033 INFO [pool-2-thread-2] o.eclipse.jetty.server.AbstractConnector Stopped ServerConnector@3f1dadeb{HTTP/1.1,[http/1.1]}{10.192.1.2:8080}
2018-03-29 17:07:25,034 INFO [pool-2-thread-2] org.eclipse.jetty.server.session Stopped scavenging<br>2018-03-29 17:07:31,669 INFO [main] org.apache.nifi.NiFi Launching NiFi...
2018-03-29 17:07:31,939 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Determined default nifi.properties path to be '/opt/blah/nifi/./conf/nifi.properties'
2018-03-29 17:07:31,943 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loaded 145 properties from /opt/blah/nifi/./conf/nifi.properties
2018-03-29 17:07:31,953 INFO [main] org.apache.nifi.NiFi Loaded 145 properties
2018-03-29 17:07:31,961 INFO [main] org.apache.nifi.BootstrapListener Started Bootstrap Listener, Listening for incoming requests on port 36566
2018-03-29 17:07:31,972 INFO [main] org.apache.nifi.BootstrapListener Successfully initiated communication with Bootstrap
5 REPLIES 5

Re: NiFi nodes restart randomly, no errors

Can you please share the complete log? A few lines of log doesn't help much.

Re: NiFi nodes restart randomly, no errors

Mentor

@Praveen Bathala
Can you also share your nifi.properties file and the nifi-app.log

Re: NiFi nodes restart randomly, no errors

Super Guru

how much RAM do you have, can you post the bootstrap and nifi.properties. is this HDF? NiFi? what versions? do you have clustering on? can they always reach each other? what kind of networking? what OS? what kind of hosting - cloud? on-premise? what kind of hardware?

this sounds like a hardware issue. I have never seen this in a hundred different installations.

Re: NiFi nodes restart randomly, no errors

Master Guru
@Praveen Bathala

The interesting line here is:

2018-03-2917:07:25,029 INFO [pool-2-thread-2] org.apache.nifi.BootstrapListenerReceived SHUTDOWN request fromBootstrap

The NIFi service was stopped and started by request of the NiFi bootstrap process.

What did you see in the nifi-bootstrap.log around the same timeframe?

Re: NiFi nodes restart randomly, no errors

Thanks for the support, I have been digging deep to figure out the issue myself and realized that our chef recipe had a bug which kept restarting NiFi every time it runs, this was reason that it restarts exactly at same time and I didn't realize it was due to external factor like chef.

Don't have an account?
Coming from Hortonworks? Activate your account here