Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

Ambari console showing wrong regionserver status.

Explorer

Ambari is showing hbase regionservers in a 'Decommissioning' status. When the regionservers are started it goes to the green checkmark (Installed status), then a few seconds later it goes to decommissioning status. The regionservers are up and appear to be running ok.

./hbase zkcli, ls /hbase-unsecure/draining and nothing is showing

./hbase-unsecure/rs and the regionservers are showing

The ambari database hostcomponentstate table shows regionservers 'STARTED'

I don't see anything in the ambari-agent log yet.

The ambari rest api shows desireds_state : STARTED for regionservers

How is the component status being set in the ambari console and how do I resolve this?

1 ACCEPTED SOLUTION

Hi @Jack Madden ,

Sorry for the delay in response. I was analyzing deep in code why is this happening and it seems like its a BUG in javascript that it shows service is decommissioning stage when the HBASE region server is already started stage.

This looks like a javascript bug to me,

we can easily workaround this by fixing in app.js

Go through this below steps and let me know if it worked for you

Steps

1) go to ambari-server and navigate to below folder and take a app.js backup

 [root@asn1 ~]# cd /usr/lib/ambari-server/web/javascripts/
 [root@asn1 javascripts]# ls -lh
total 29M
-rw-r--r--. 1 root root 8.6M Dec 17 09:33 app.js
-rw-r--r--. 1 root root 2.5M Dec  7 18:32 vendor.js
[root@asn1 javascripts]# cp app.js app.js_backup_original

2) edit the app.js file

[root@asn1 javascripts]# vi app.js

findout the line :

this.setStatusAs('RS_DECOMMISSIONED');

and comment it

//this.setStatusAs('RS_DECOMMISSIONED');

3) save the file and Hard reload the ambari-web browser(open in incognito tab to remove the possbility of cached app.js)

Please share the feedback if it worked or not.

Please login and accept the answer if it did

Note : The following changes are based on my inspection and analysis. this BUG is not yet fixed in ambari, you can revert back the app.js after the change incases you wish for valid fix. i will raise apache bug for this and fix later.

View solution in original post

11 REPLIES 11

Hi @Jack Madden,

If the decommissioning process is completed and just ambar is showing wrong information in ui , then can you try disable maintainence mode for the region server and see if there is any luck ?

BTW the ui fetches the Status of component via CURL

curl -u admin:admin -H "X-Requested-By:ambari" -i -X GET http://<AMBARI_HOST>:8080/api/v1/clusters/<CLUSTER_NAME>/hosts/<HOSTNAME>/host_components/HBASE_REGI...

and will look for desired_admin_state , whats the output of this command for you ?

Also try to enable debug log for ambari-agent , restart ambari-agent to see whats the status of region server in heartbeat message send from ambari-agent to ambari-server.

Explorer

Partial outptut:

{
  "href" : "https://hadoop-1.nit.disa.mil:8443/api/v1/clusters/PIEE/hosts/hadoop-2.nit.disa.mil/host_components/HBASE_REGIONSERVER/?fields=*",
  "HostRoles" : {
    "cluster_name" : "PIEE",
    "component_name" : "HBASE_REGIONSERVER",
    "desired_admin_state" : "INSERVICE",
    "desired_repository_version" : "3.0.1.0-187",
    "desired_stack_id" : "HDP-3.0",
    "desired_state" : "STARTED",
    "display_name" : "RegionServer",
    "host_name" : "hadoop-2.nit.disa.mil",
    "maintenance_state" : "OFF",
    "public_host_name" : "hadoop-2.nit.disa.mil",
    "reload_configs" : false,
    "service_name" : "HBASE",
    "stale_configs" : false,
    "state" : "STARTED",
    "upgrade_state" : "NONE",
    "version" : "3.0.1.0-187",
    "actual_configs" : { }
  },

Explorer

Here is the output from ambari-agent log debug level:

DEBUG 2018-12-26 11:21:32,977 HeartbeatThread.py:99 - Heartbeat response is {u'status': u'OK', u'id': 8}

Hi @Jack Madden,

from your previous comment i see "desired_admin_state" is in "INSERVICE", but as per you it is showing a DECOMMISSIONING State . Also the CURL command tells us the host-component is not in maintenance_state .

We can try following things here to get the correct status in UI

1) try restarting ambari-server if doesnt work Take a backup of database and look out for output of following commands

2) go to database, inspect the output of the following command :

ambari=> select id,component_name,desired_State,host_id,service_name,admin_state  from hostcomponentdesiredstate where service_name='HBASE';
 id  |   component_name   | desired_state | host_id | service_name | admin_state
-----+--------------------+---------------+---------+--------------+-------------
   9 | HBASE_MASTER       | STARTED       |       6 | HBASE        |
  27 | HBASE_REGIONSERVER | STARTED       |       2 | HBASE        | INSERVICE
  28 | HBASE_REGIONSERVER | STARTED       |       5 | HBASE        | INSERVICE
  86 | HBASE_CLIENT       | INSTALLED     |       1 | HBASE        |
  53 | HBASE_CLIENT       | INSTALLED     |       2 | HBASE        |
  54 | HBASE_CLIENT       | INSTALLED     |       5 | HBASE        |
  26 | HBASE_REGIONSERVER | STARTED       |       1 | HBASE        |
 151 | HBASE_CLIENT       | INSTALLED     |       6 | HBASE        |
 152 | HBASE_REGIONSERVER | STARTED       |       6 | HBASE        | INSERVICE
(9 rows)

3) correct the admin_state column if required. please remember to set it as NULL , example command :

ambari=> update hostcomponentdesiredstate set admin_state=NULL where id=<ID>;

4) restart ambari to see if there is any luck.

Note : Please take proper backup of the database and be 1000% sure of what command you are executing, make sure ambari-server is stopped before performing any edit to the database.If it's a production cluster always contact hortonworks support .

Hope this helps you troubleshooting more on this regard.

Please accept the answer if it helped.

Explorer

It displays started and a green check mark for a few seconds then eventually goes to a decommissioning status. It's as if there's a process that's checking something and changing the status

Explorer

I changed the admin_state to NULL and restarted ambari. The regionserver status still shows 'Decommissioning'. The rest api still shows desired_admin_state" : "INSERVICE.

Explorer
id | component_name | desired_state | host_id | service_name | admin_state 
-----+----------------------+---------------+---------+--------------+------------- 
154 | PHOENIX_QUERY_SERVER | STARTED | 1 | HBASE | INSERVICE 
155 | PHOENIX_QUERY_SERVER | STARTED | 2 | HBASE | INSERVICE 
159 | HBASE_CLIENT | INSTALLED | 2 | HBASE | 
157 | HBASE_CLIENT | INSTALLED | 1 | HBASE | 
158 | HBASE_CLIENT | INSTALLED | 3 | HBASE | 
156 | PHOENIX_QUERY_SERVER | STARTED | 3 | HBASE | INSERVICE 
152 | HBASE_REGIONSERVER | STARTED | 3 | HBASE | 
153 | HBASE_REGIONSERVER | STARTED | 1 | HBASE | 
151 | HBASE_MASTER | STARTED | 2 | HBASE |

  "href" : "https://hadoop-1.nit.disa.mil:8443/api/v1/clusters/PIEE/hosts/hadoop-2.nit.disa.mil/host_components/HBASE_REGIONSERVER/?fields=*",
  "HostRoles" : {
    "cluster_name" : "PIEE",
    "component_name" : "HBASE_REGIONSERVER",
    "desired_admin_state" : "INSERVICE",
    "desired_repository_version" : "3.0.1.0-187",
    "desired_stack_id" : "HDP-3.0",
    "desired_state" : "STARTED",
    "display_name" : "RegionServer",
    "host_name" : "hadoop-2.nit.disa.mil",
    "maintenance_state" : "OFF",
    "public_host_name" : "hadoop-2.nit.disa.mil",
    "reload_configs" : false,
    "service_name" : "HBASE",
    "stale_configs" : false,
    "state" : "STARTED",
    "upgrade_state" : "NONE",
    "version" : "3.0.1.0-187",
    "actual_configs" : { }
  }

Explorer

Hi,

I updated the hostcomponentdesiredstate table but the regionserver component is still showing 'Decommissioning'.

ambari=> select id,component_name,desired_State,host_id,service_name,admin_state  from hostcomponentdesiredstate where service_name='HBASE';
 id  |    component_name    | desired_state | host_id | service_name | admin_state
-----+----------------------+---------------+---------+--------------+-------------
 154 | PHOENIX_QUERY_SERVER | STARTED       |       1 | HBASE        | INSERVICE
 155 | PHOENIX_QUERY_SERVER | STARTED       |       2 | HBASE        | INSERVICE
 159 | HBASE_CLIENT         | INSTALLED     |       2 | HBASE        |
 157 | HBASE_CLIENT         | INSTALLED     |       1 | HBASE        |
 158 | HBASE_CLIENT         | INSTALLED     |       3 | HBASE        |
 156 | PHOENIX_QUERY_SERVER | STARTED       |       3 | HBASE        | INSERVICE
 152 | HBASE_REGIONSERVER   | STARTED       |       3 | HBASE        |
 153 | HBASE_REGIONSERVER   | STARTED       |       1 | HBASE        |
 151 | HBASE_MASTER         | STARTED       |       2 | HBASE        |
(9 rows)


The desired_admin_state still says 'inservice' in the rest api.

 "HostRoles" : {
    "cluster_name" : "PIEE",
    "component_name" : "HBASE_REGIONSERVER",
    "desired_admin_state" : "INSERVICE",
    "desired_repository_version" : "3.0.1.0-187",
    "desired_stack_id" : "HDP-3.0",
    "desired_state" : "STARTED",
    "display_name" : "RegionServer",
    "host_name" : "hadoop-2.nit.disa.mil",
    "maintenance_state" : "OFF",
    "public_host_name" : "hadoop-2.nit.disa.mil",
    "reload_configs" : false,
    "service_name" : "HBASE",
    "stale_configs" : false,
    "state" : "STARTED",
    "upgrade_state" : "NONE",
    "version" : "3.0.1.0-187",
    "actual_configs" : { }
  },

Hi @Jack Madden ,

Sorry for the delay in response. I was analyzing deep in code why is this happening and it seems like its a BUG in javascript that it shows service is decommissioning stage when the HBASE region server is already started stage.

This looks like a javascript bug to me,

we can easily workaround this by fixing in app.js

Go through this below steps and let me know if it worked for you

Steps

1) go to ambari-server and navigate to below folder and take a app.js backup

 [root@asn1 ~]# cd /usr/lib/ambari-server/web/javascripts/
 [root@asn1 javascripts]# ls -lh
total 29M
-rw-r--r--. 1 root root 8.6M Dec 17 09:33 app.js
-rw-r--r--. 1 root root 2.5M Dec  7 18:32 vendor.js
[root@asn1 javascripts]# cp app.js app.js_backup_original

2) edit the app.js file

[root@asn1 javascripts]# vi app.js

findout the line :

this.setStatusAs('RS_DECOMMISSIONED');

and comment it

//this.setStatusAs('RS_DECOMMISSIONED');

3) save the file and Hard reload the ambari-web browser(open in incognito tab to remove the possbility of cached app.js)

Please share the feedback if it worked or not.

Please login and accept the answer if it did

Note : The following changes are based on my inspection and analysis. this BUG is not yet fixed in ambari, you can revert back the app.js after the change incases you wish for valid fix. i will raise apache bug for this and fix later.

Explorer

Thanks for digging into this and giving temporary workaround. I changed app.js and the regionserver shows started (green checkmark). The actions available list on the regionserver now are stop, turn on maintenance mode, and delete is disabled.

} else if (isInServiceDesired) {
//this.setStatusAs('RS_DECOMMISSIONED');

this.setSatusAs('INSERVICE'); sets the regionserver to the correct status and correct actions list.

} else if (isInServiceDesired) {
//this.setStatusAs('RS_DECOMMISSIONED');
this.setStatusAs('INSERVICE');
}

Thanks again for looking at this. How would I track the apache bug submission.

Hey Jack,

Very much glad that it worked.

I have created an apache jira on this regard : https://issues.apache.org/jira/browse/AMBARI-25075

you can be yourself a jira watcher to this JIRA.