question Re: What is best way to reboot machines in the Hadoop Cluster ? in Archives of Support Questions (Read Only)

What is best way to reboot machines in the Hadoop Cluster ?

panda_anilkumar — Mon, 18 Jan 2016 17:18:53 GMT

We have a recently build 5 node HDP cluster, it is not in HA mode. Some time in the future there will be an requirement from the Unix team to apply patches/server maintenance. which would require rebooting the server machines. What is the best way to do it ?

I plan to do the following :

1) Shut down all services using Ambari.

2) Shutdown ambari-agents on all nodes.

3) Shutdown ambari-server.

4) Reboot all nodes as required .

5) Restart ambari-server, agents and services in that order.

Is this the correct sequence ? or am I missing anything .

Re: What is best way to reboot machines in the Hadoop Cluster ?

jstraub — Mon, 18 Jan 2016 17:38:05 GMT

Sounds good.

Make sure all services did stop successfully and Namenode is in safemode before you reboot.

Note: Before you patch your system make sure your backups are up2date. See this https://community.hortonworks.com/questions/4135/what-to-backup-and-how-only-metadata-not-data.html

Re: What is best way to reboot machines in the Hadoop Cluster ?

orenault — Mon, 18 Jan 2016 17:43:16 GMT

Switching off ambari-agent is a bit overkill as it will be done by sysinit upon system reboot. In the same way, it will be restarted automatically.

For your datanodes, you may want to do one at a time to limit the downtime.

Re: What is best way to reboot machines in the Hadoop Cluster ?

pankaj_singh — Mon, 18 Jan 2016 17:43:45 GMT

@Anilkumar Panda

Also have a look at https://community.hortonworks.com/questions/4940/hdp-os-upgradepatching-best-practices.html

Re: What is best way to reboot machines in the Hadoop Cluster ?

nsabharwal — Mon, 18 Jan 2016 20:14:08 GMT

@Anilkumar Panda

I like your approach. This is not different from typical RDBMS or any other application stack.

1) Shut down all services using Ambari.

you can use rest API to shutdown services if your like link (You can create a shell script to call stop function on the services that you have installed)

2) Shutdown ambari-agents on all nodes.

3) Shutdown ambari-server.

4) Reboot all nodes as required .

5) Restart ambari-server, agents and services in that order. (chkdconfig on ambari-server and agent helps to bring up services on reboot)

Re: What is best way to reboot machines in the Hadoop Cluster ?

aervits — Thu, 04 Feb 2016 10:48:22 GMT

@Anilkumar Panda are you still having issues with this? Can you accept best answer or provide your own solution?