question Re: NiFi - restarting a node gracefully in Support Questions

NiFi - restarting a node gracefully

albert_phu — Wed, 21 Jun 2017 21:57:19 GMT

I have a cluster of lets say 3 NiFi nodes and I would like to make a configuration change, what are the steps to do a graceful restart of each node one by one?

using "service nifi stop" and "service nifi start"

I always get the following output -

2017-06-20 23:19:04,097 INFO [main] org.apache.nifi.bootstrap.Command 
Apache NiFi has accepted the Shutdown Command and is shutting down now
2017-06-20 23:19:04,141 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:06,145 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:08,149 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:10,153 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:12,157 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:14,162 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:16,167 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:18,171 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:20,176 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:22,180 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:24,184 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:24,200 WARN [main] org.apache.nifi.bootstrap.Command 
NiFi has not finished shutting down after 20 seconds. Killing process.
2017-06-20 23:19:24,218 INFO [main] org.apache.nifi.bootstrap.Command 
NiFi has finished shutting down.

I would rather not have NiFi killed if it is in the middle of something, is this to be expected?

Re: NiFi - restarting a node gracefully

bbende — Thu, 22 Jun 2017 01:44:09 GMT

Do you have any custom components deployed?

Re: NiFi - restarting a node gracefully

JoeWitt — Thu, 22 Jun 2017 03:07:54 GMT

Any component, custom or not, which is not responding in a timely manner to lifecycle calls such as when it is unscheduled will do this. I've seen it quite a bit lately as well. We should consider listing thread titles or something that have it as that will help spot the culprit pretty quickly.

Re: NiFi - restarting a node gracefully

albert_phu — Thu, 22 Jun 2017 05:25:57 GMT

I have no custom modules deployed.

Re: NiFi - restarting a node gracefully

albert_phu — Fri, 23 Jun 2017 06:32:08 GMT

currently there is no way to see what is holding it up?

Re: NiFi - restarting a node gracefully

mburgess — Sat, 24 Jun 2017 00:23:37 GMT

You can try a thread dump (with jstack or nifi.sh dump) while it is waiting to shut down, you may be able to spot the culprit in the output.

Re: NiFi - restarting a node gracefully

fp — Thu, 12 Apr 2018 20:37:44 GMT

I have a similar problem and I added a thread dump in this separate question that hasn't been answered yet:

https://community.hortonworks.com/questions/182912/nifi-is-failing-to-shutdown-gracefully-how-to-debu.html

Any help would be greatly appreciated.