What is best way to reboot machines in the Hadoop Cluster ?

[#1]

We have a recently build 5 node HDP cluster, it is not in HA mode. Some time in the future there will be an requirement from the Unix team to apply patches/server maintenance. which would require rebooting the server machines. What is the best way to do it ?

I plan to do the following :

1) Shut down all services using Ambari.

2) Shutdown ambari-agents on all nodes.

3) Shutdown ambari-server.

4) Reboot all nodes as required .

5) Restart ambari-server, agents and services in that order.

Is this the correct sequence ? or am I missing anything .

[#2]

Sounds good.

Make sure all services did stop successfully and Namenode is in safemode before you reboot.

Note: Before you patch your system make sure your backups are up2date. See this https://community.hortonworks.com/questions/4135/what-to-backup-and-how-only-metadata-not-data.html

[#3]

Switching off ambari-agent is a bit overkill as it will be done by sysinit upon system reboot. In the same way, it will be restarted automatically.

For your datanodes, you may want to do one at a time to limit the downtime.

[#4]

@Anilkumar Panda

Also have a look at https://community.hortonworks.com/questions/4940/hdp-os-upgradepatching-best-practices.html

[#5 · Accepted solution]

@Anilkumar Panda

I like your approach. This is not different from typical RDBMS or any other application stack.

1) Shut down all services using Ambari.

you can use rest API to shutdown services if your like link (You can create a shell script to call stop function on the services that you have installed)

2) Shutdown ambari-agents on all nodes.

3) Shutdown ambari-server.

4) Reboot all nodes as required .

5) Restart ambari-server, agents and services in that order. (chkdconfig on ambari-server and agent helps to bring up services on reboot)

[#6]

@Anilkumar Panda are you still having issues with this? Can you accept best answer or provide your own solution?