Ambari console showing wrong regionserver status.

[#1]

Ambari is showing hbase regionservers in a 'Decommissioning' status. When the regionservers are started it goes to the green checkmark (Installed status), then a few seconds later it goes to decommissioning status. The regionservers are up and appear to be running ok.

./hbase zkcli, ls /hbase-unsecure/draining and nothing is showing

./hbase-unsecure/rs and the regionservers are showing

The ambari database hostcomponentstate table shows regionservers 'STARTED'

I don't see anything in the ambari-agent log yet.

The ambari rest api shows desireds_state : STARTED for regionservers

How is the component status being set in the ambari console and how do I resolve this?

[#2]

Hi @Jack Madden,

If the decommissioning process is completed and just ambar is showing wrong information in ui , then can you try disable maintainence mode for the region server and see if there is any luck ?

BTW the ui fetches the Status of component via CURL

curl -u admin:admin -H "X-Requested-By:ambari" -i -X GET http://<AMBARI_HOST>:8080/api/v1/clusters/<CLUSTER_NAME>/hosts/<HOSTNAME>/host_components/HBASE_REGI...

and will look for desired_admin_state , whats the output of this command for you ?

Also try to enable debug log for ambari-agent , restart ambari-agent to see whats the status of region server in heartbeat message send from ambari-agent to ambari-server.

[#3]

Partial outptut:

{
  "href" : "https://hadoop-1.nit.disa.mil:8443/api/v1/clusters/PIEE/hosts/hadoop-2.nit.disa.mil/host_components/HBASE_REGIONSERVER/?fields=*",
  "HostRoles" : {
    "cluster_name" : "PIEE",
    "component_name" : "HBASE_REGIONSERVER",
    "desired_admin_state" : "INSERVICE",
    "desired_repository_version" : "3.0.1.0-187",
    "desired_stack_id" : "HDP-3.0",
    "desired_state" : "STARTED",
    "display_name" : "RegionServer",
    "host_name" : "hadoop-2.nit.disa.mil",
    "maintenance_state" : "OFF",
    "public_host_name" : "hadoop-2.nit.disa.mil",
    "reload_configs" : false,
    "service_name" : "HBASE",
    "stale_configs" : false,
    "state" : "STARTED",
    "upgrade_state" : "NONE",
    "version" : "3.0.1.0-187",
    "actual_configs" : { }
  },

[#4]

Here is the output from ambari-agent log debug level:

DEBUG 2018-12-26 11:21:32,977 HeartbeatThread.py:99 - Heartbeat response is {u'status': u'OK', u'id': 8}

[#5]

Hi @Jack Madden,

from your previous comment i see "desired_admin_state" is in "INSERVICE", but as per you it is showing a DECOMMISSIONING State . Also the CURL command tells us the host-component is not in maintenance_state .

We can try following things here to get the correct status in UI

1) try restarting ambari-server if doesnt work Take a backup of database and look out for output of following commands

2) go to database, inspect the output of the following command :

ambari=> select id,component_name,desired_State,host_id,service_name,admin_state  from hostcomponentdesiredstate where service_name='HBASE';
 id  |   component_name   | desired_state | host_id | service_name | admin_state
-----+--------------------+---------------+---------+--------------+-------------
   9 | HBASE_MASTER       | STARTED       |       6 | HBASE        |
  27 | HBASE_REGIONSERVER | STARTED       |       2 | HBASE        | INSERVICE
  28 | HBASE_REGIONSERVER | STARTED       |       5 | HBASE        | INSERVICE
  86 | HBASE_CLIENT       | INSTALLED     |       1 | HBASE        |
  53 | HBASE_CLIENT       | INSTALLED     |       2 | HBASE        |
  54 | HBASE_CLIENT       | INSTALLED     |       5 | HBASE        |
  26 | HBASE_REGIONSERVER | STARTED       |       1 | HBASE        |
 151 | HBASE_CLIENT       | INSTALLED     |       6 | HBASE        |
 152 | HBASE_REGIONSERVER | STARTED       |       6 | HBASE        | INSERVICE
(9 rows)

3) correct the admin_state column if required. please remember to set it as NULL , example command :

ambari=> update hostcomponentdesiredstate set admin_state=NULL where id=<ID>;

4) restart ambari to see if there is any luck.

Note : Please take proper backup of the database and be 1000% sure of what command you are executing, make sure ambari-server is stopped before performing any edit to the database.If it's a production cluster always contact hortonworks support .

Hope this helps you troubleshooting more on this regard.

Please accept the answer if it helped.

[#6]

It displays started and a green check mark for a few seconds then eventually goes to a decommissioning status. It's as if there's a process that's checking something and changing the status

[#7]

I changed the admin_state to NULL and restarted ambari. The regionserver status still shows 'Decommissioning'. The rest api still shows desired_admin_state" : "INSERVICE.

[#8]

id | component_name | desired_state | host_id | service_name | admin_state 
-----+----------------------+---------------+---------+--------------+------------- 
154 | PHOENIX_QUERY_SERVER | STARTED | 1 | HBASE | INSERVICE 
155 | PHOENIX_QUERY_SERVER | STARTED | 2 | HBASE | INSERVICE 
159 | HBASE_CLIENT | INSTALLED | 2 | HBASE | 
157 | HBASE_CLIENT | INSTALLED | 1 | HBASE | 
158 | HBASE_CLIENT | INSTALLED | 3 | HBASE | 
156 | PHOENIX_QUERY_SERVER | STARTED | 3 | HBASE | INSERVICE 
152 | HBASE_REGIONSERVER | STARTED | 3 | HBASE | 
153 | HBASE_REGIONSERVER | STARTED | 1 | HBASE | 
151 | HBASE_MASTER | STARTED | 2 | HBASE |

  "href" : "https://hadoop-1.nit.disa.mil:8443/api/v1/clusters/PIEE/hosts/hadoop-2.nit.disa.mil/host_components/HBASE_REGIONSERVER/?fields=*",
  "HostRoles" : {
    "cluster_name" : "PIEE",
    "component_name" : "HBASE_REGIONSERVER",
    "desired_admin_state" : "INSERVICE",
    "desired_repository_version" : "3.0.1.0-187",
    "desired_stack_id" : "HDP-3.0",
    "desired_state" : "STARTED",
    "display_name" : "RegionServer",
    "host_name" : "hadoop-2.nit.disa.mil",
    "maintenance_state" : "OFF",
    "public_host_name" : "hadoop-2.nit.disa.mil",
    "reload_configs" : false,
    "service_name" : "HBASE",
    "stale_configs" : false,
    "state" : "STARTED",
    "upgrade_state" : "NONE",
    "version" : "3.0.1.0-187",
    "actual_configs" : { }
  }

[#9]

Hi,

I updated the hostcomponentdesiredstate table but the regionserver component is still showing 'Decommissioning'.

ambari=> select id,component_name,desired_State,host_id,service_name,admin_state  from hostcomponentdesiredstate where service_name='HBASE';
 id  |    component_name    | desired_state | host_id | service_name | admin_state
-----+----------------------+---------------+---------+--------------+-------------
 154 | PHOENIX_QUERY_SERVER | STARTED       |       1 | HBASE        | INSERVICE
 155 | PHOENIX_QUERY_SERVER | STARTED       |       2 | HBASE        | INSERVICE
 159 | HBASE_CLIENT         | INSTALLED     |       2 | HBASE        |
 157 | HBASE_CLIENT         | INSTALLED     |       1 | HBASE        |
 158 | HBASE_CLIENT         | INSTALLED     |       3 | HBASE        |
 156 | PHOENIX_QUERY_SERVER | STARTED       |       3 | HBASE        | INSERVICE
 152 | HBASE_REGIONSERVER   | STARTED       |       3 | HBASE        |
 153 | HBASE_REGIONSERVER   | STARTED       |       1 | HBASE        |
 151 | HBASE_MASTER         | STARTED       |       2 | HBASE        |
(9 rows)

The desired_admin_state still says 'inservice' in the rest api.

 "HostRoles" : {
    "cluster_name" : "PIEE",
    "component_name" : "HBASE_REGIONSERVER",
    "desired_admin_state" : "INSERVICE",
    "desired_repository_version" : "3.0.1.0-187",
    "desired_stack_id" : "HDP-3.0",
    "desired_state" : "STARTED",
    "display_name" : "RegionServer",
    "host_name" : "hadoop-2.nit.disa.mil",
    "maintenance_state" : "OFF",
    "public_host_name" : "hadoop-2.nit.disa.mil",
    "reload_configs" : false,
    "service_name" : "HBASE",
    "stale_configs" : false,
    "state" : "STARTED",
    "upgrade_state" : "NONE",
    "version" : "3.0.1.0-187",
    "actual_configs" : { }
  },

[#10 · Accepted solution]

Hi @Jack Madden ,

Sorry for the delay in response. I was analyzing deep in code why is this happening and it seems like its a BUG in javascript that it shows service is decommissioning stage when the HBASE region server is already started stage.

This looks like a javascript bug to me,

we can easily workaround this by fixing in app.js

Go through this below steps and let me know if it worked for you

Steps

1) go to ambari-server and navigate to below folder and take a app.js backup

 [root@asn1 ~]# cd /usr/lib/ambari-server/web/javascripts/
 [root@asn1 javascripts]# ls -lh
total 29M
-rw-r--r--. 1 root root 8.6M Dec 17 09:33 app.js
-rw-r--r--. 1 root root 2.5M Dec  7 18:32 vendor.js
[root@asn1 javascripts]# cp app.js app.js_backup_original

2) edit the app.js file

[root@asn1 javascripts]# vi app.js

findout the line :

this.setStatusAs('RS_DECOMMISSIONED');

and comment it

//this.setStatusAs('RS_DECOMMISSIONED');

3) save the file and Hard reload the ambari-web browser(open in incognito tab to remove the possbility of cached app.js)

Please share the feedback if it worked or not.

Please login and accept the answer if it did

Note : The following changes are based on my inspection and analysis. this BUG is not yet fixed in ambari, you can revert back the app.js after the change incases you wish for valid fix. i will raise apache bug for this and fix later.