NiFi - restarting a node gracefully

[#1]

I have a cluster of lets say 3 NiFi nodes and I would like to make a configuration change, what are the steps to do a graceful restart of each node one by one?

using "service nifi stop" and "service nifi start"

I always get the following output -

2017-06-20 23:19:04,097 INFO [main] org.apache.nifi.bootstrap.Command 
Apache NiFi has accepted the Shutdown Command and is shutting down now
2017-06-20 23:19:04,141 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:06,145 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:08,149 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:10,153 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:12,157 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:14,162 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:16,167 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:18,171 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:20,176 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:22,180 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:24,184 INFO [main] org.apache.nifi.bootstrap.Command 
Waiting for Apache NiFi to finish shutting down...
2017-06-20 23:19:24,200 WARN [main] org.apache.nifi.bootstrap.Command 
NiFi has not finished shutting down after 20 seconds. Killing process.
2017-06-20 23:19:24,218 INFO [main] org.apache.nifi.bootstrap.Command 
NiFi has finished shutting down.

I would rather not have NiFi killed if it is in the middle of something, is this to be expected?

[#2]

Do you have any custom components deployed?

[#3 · Accepted solution]

Any component, custom or not, which is not responding in a timely manner to lifecycle calls such as when it is unscheduled will do this. I've seen it quite a bit lately as well. We should consider listing thread titles or something that have it as that will help spot the culprit pretty quickly.

[#4]

currently there is no way to see what is holding it up?

[#5]

You can try a thread dump (with jstack or nifi.sh dump) while it is waiting to shut down, you may be able to spot the culprit in the output.

[#6]

I have a similar problem and I added a thread dump in this separate question that hasn't been answered yet:

https://community.hortonworks.com/questions/182912/nifi-is-failing-to-shutdown-gracefully-how-to-deb...

Any help would be greatly appreciated.

[#7]

I have no custom modules deployed.