Deleting a dead host from Ambari

mikelok — Sun, 10 Nov 2019 12:28:07 GMT

Hi all,

I'm have a 9 host cluster on HDP 2.6.5. Recently, a host went down and had to be rebuilt. It was hosting ZK server, DataNode and YARN server. Can I safely delete the host from Ambari web UI as I'm unable to turn off the services?

Thanks.

Re: Deleting a dead host from Ambari

Shelton — Mon, 11 Nov 2019 20:48:44 GMT

@mikelok

Before you embark on that there are a couple of questions? Have you moved or recreated the zookeeper , YARN, and Datanode on another node?

You should be having 3 zookeepers at least. and how about your YARN server? Usually, when a host crashed and stops sending the heartbeat after a period of time it's excluded from the health nodes.

Procedure

1. Decommission DataNodes:

From the node hosting the NameNode, edit the $HADOOP_CONF_DIR/dfs. exclude file by adding the list of DataNode hostnames, separated by a newline character.
Update the NameNode with the new set of excluded DataNodes. Run the following command from the NameNode machine:
```
# su $HDFS_USER
$ hdfs dfsadmin -refreshNodes
```
$HDFS_USER is the user that owns the HDFS services, which is usually hdfs.
Open the NameNode web interface, and go to the DataNodes page:
http://<abc.my_namenode.com>:50070. Verify that the state is changed to Decommission in Progress for the DataNodes that are being decommissioned.
Shut down the decommissioned nodes when all of the DataNodes are decommissioned. All of the blocks will already be replicated.
If you use a dfs.include file on your cluster, remove the decommissioned nodes from that file on the NameNode host machine. Then refresh the nodes on that machine:
```
# su $HDFS_USER
$ hdfs dfsadmin -refreshNodes
```
If no dfs.include is used, all DataNodes are considered included in the cluster, unless a node exists in a $HADOOP_CONF_DIR/dfs.exclude file.

You can also use the Ambari REST API to achieve that here is a reference

https://cwiki.apache.org/confluence/display/AMBARI/Using+APIs+to+delete+a+service+or+all+host+components+on+a+host

Hope that helps

Re: Deleting a dead host from Ambari

mikelok — Sun, 17 Nov 2019 05:42:12 GMT

Hi Shelton,

The steps you provided worked perfectly.

Thanks!

question Deleting a dead host from Ambari in Support Questions

Deleting a dead host from Ambari

Re: Deleting a dead host from Ambari

Procedure

Re: Deleting a dead host from Ambari