Support Questions

Find answers, ask questions, and share your expertise

NiFi - restarting a node gracefully

avatar
New Contributor

I have a cluster of lets say 3 NiFi nodes and I would like to make a configuration change, what are the steps to do a graceful restart of each node one by one?

using "service nifi stop" and "service nifi start"

I always get the following output -

2017-06-20 23:19:04,097 INFO [main] org.apache.nifi.bootstrap.Command 
Apache NiFi has accepted the Shutdown Command and is shutting down now
2017-06-20 23:19:04,141 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:06,145 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:08,149 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:10,153 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:12,157 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:14,162 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:16,167 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:18,171 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:20,176 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:22,180 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:24,184 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:24,200 WARN [main] org.apache.nifi.bootstrap.Command 
NiFi has not finished shutting down after 20 seconds. Killing process.
2017-06-20 23:19:24,218 INFO [main] org.apache.nifi.bootstrap.Command 
NiFi has finished shutting down.

I would rather not have NiFi killed if it is in the middle of something, is this to be expected?

1 ACCEPTED SOLUTION

avatar

Any component, custom or not, which is not responding in a timely manner to lifecycle calls such as when it is unscheduled will do this. I've seen it quite a bit lately as well. We should consider listing thread titles or something that have it as that will help spot the culprit pretty quickly.

View solution in original post

6 REPLIES 6

avatar
Master Guru

Do you have any custom components deployed?

avatar

Any component, custom or not, which is not responding in a timely manner to lifecycle calls such as when it is unscheduled will do this. I've seen it quite a bit lately as well. We should consider listing thread titles or something that have it as that will help spot the culprit pretty quickly.

avatar
New Contributor

currently there is no way to see what is holding it up?

avatar
Master Guru

You can try a thread dump (with jstack or nifi.sh dump) while it is waiting to shut down, you may be able to spot the culprit in the output.

avatar
New Contributor

I have a similar problem and I added a thread dump in this separate question that hasn't been answered yet:

https://community.hortonworks.com/questions/182912/nifi-is-failing-to-shutdown-gracefully-how-to-deb...

Any help would be greatly appreciated.

avatar
New Contributor

I have no custom modules deployed.