Support Questions

Find answers, ask questions, and share your expertise

Decommission of Region Server

Contributor

is it possible to Perform a graceful stop decommission using Ambari.

3 REPLIES 3

Mentor

@Dhiraj

Graceful stop YES decommissioning NO

If your region server was installed successfully you can remove it using the Ambari REST API, make sure the region server is stopped before attempting their removal see below example, replace the placeholders with correct values

curl -u $USER:$PASS -i -H 'X-Requested-By: ambari' -X DELETE http://$AMBARI_HOST:$AMBARI_PORT/api/v1/clusters/<cluster-name>/services/<service-name>;

Hope that helps

@Dhiraj

In addition to @Geoffrey Shelton Okot if you want to talk care from commandline

Node Decommission

You can stop an individual RegionServer by running the following script in the HBase directory on the particular node:

$ ./bin/hbase-daemon.sh stop regionserver

The RegionServer will first close all regions and then shut itself down. On shutdown, the RegionServer's ephemeral node in ZooKeeper will expire. The master will notice the RegionServer gone and will treat it as a 'crashed' server; it will reassign the nodes the RegionServer was carrying.

Disable the Load Balancer before Decommissioning a node

If the load balancer runs while a node is shutting down, then there could be contention between the Load Balancer and the Master's recovery of the just decommissioned RegionServer. Avoid any problems by disabling the balancer first.

A downside to the above stop of a RegionServer is that regions could be offline for a good period of time. Regions are closed in order. If many regions on the server, the first region to close may not be back online until all regions close and after the master notices the RegionServer's znode gone. In Apache HBase 0.90.2, we added facility for having a node gradually shed its load and then shutdown itself down.

To decommission a loaded RegionServer, run the following:

$ ./bin/graceful_stop.sh HOSTNAME

where HOSTNAME is the host carrying the RegionServer you would decommission.

On HOSTNAME

The HOSTNAME passed to graceful_stop.sh must match the hostname that hbase is using to identify RegionServers. Check the list of RegionServers in the master UI for how HBase is referring to servers. Its usually hostname but can also be FQDN. Whatever HBase is using, this is what you should pass the graceful_stop.sh decommission script. If you pass IPs, the script is not yet smart enough to make a hostname (or FQDN) of it and so it will fail when it checks if server is currently running; the graceful unloading of regions will not run.

The graceful_stop.sh script will move the regions off the decommissioned RegionServer one at a time to minimize region churn. It will verify the region deployed in the new location before it will moves the next region and so on until the decommissioned server is carrying zero regions. At this point, the graceful_stop.sh tells the RegionServer stop. The master will at this point notice the RegionServer gone but all regions will have already been redeployed and because the RegionServer went down cleanly, there will be no WAL logs to split.

Hope this helps you.

Contributor

@Sridhar Reddy hbase-daemon.sh is available in given location, But I don't see graceful_stop.sh?

Hbase Version is 1.1.2 HDP-2.4.3.0

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.