Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Ambari console showing wrong regionserver status.

avatar
Explorer

Ambari is showing hbase regionservers in a 'Decommissioning' status. When the regionservers are started it goes to the green checkmark (Installed status), then a few seconds later it goes to decommissioning status. The regionservers are up and appear to be running ok.

./hbase zkcli, ls /hbase-unsecure/draining and nothing is showing

./hbase-unsecure/rs and the regionservers are showing

The ambari database hostcomponentstate table shows regionservers 'STARTED'

I don't see anything in the ambari-agent log yet.

The ambari rest api shows desireds_state : STARTED for regionservers

How is the component status being set in the ambari console and how do I resolve this?

1 ACCEPTED SOLUTION

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
11 REPLIES 11

avatar

Hi @Jack Madden,

If the decommissioning process is completed and just ambar is showing wrong information in ui , then can you try disable maintainence mode for the region server and see if there is any luck ?

BTW the ui fetches the Status of component via CURL

curl -u admin:admin -H "X-Requested-By:ambari" -i -X GET http://<AMBARI_HOST>:8080/api/v1/clusters/<CLUSTER_NAME>/hosts/<HOSTNAME>/host_components/HBASE_REGI...

and will look for desired_admin_state , whats the output of this command for you ?

Also try to enable debug log for ambari-agent , restart ambari-agent to see whats the status of region server in heartbeat message send from ambari-agent to ambari-server.

avatar
Explorer

Partial outptut:

{
  "href" : "https://hadoop-1.nit.disa.mil:8443/api/v1/clusters/PIEE/hosts/hadoop-2.nit.disa.mil/host_components/HBASE_REGIONSERVER/?fields=*",
  "HostRoles" : {
    "cluster_name" : "PIEE",
    "component_name" : "HBASE_REGIONSERVER",
    "desired_admin_state" : "INSERVICE",
    "desired_repository_version" : "3.0.1.0-187",
    "desired_stack_id" : "HDP-3.0",
    "desired_state" : "STARTED",
    "display_name" : "RegionServer",
    "host_name" : "hadoop-2.nit.disa.mil",
    "maintenance_state" : "OFF",
    "public_host_name" : "hadoop-2.nit.disa.mil",
    "reload_configs" : false,
    "service_name" : "HBASE",
    "stale_configs" : false,
    "state" : "STARTED",
    "upgrade_state" : "NONE",
    "version" : "3.0.1.0-187",
    "actual_configs" : { }
  },

avatar
Explorer

Here is the output from ambari-agent log debug level:

DEBUG 2018-12-26 11:21:32,977 HeartbeatThread.py:99 - Heartbeat response is {u'status': u'OK', u'id': 8}

avatar

Hi @Jack Madden,

from your previous comment i see "desired_admin_state" is in "INSERVICE", but as per you it is showing a DECOMMISSIONING State . Also the CURL command tells us the host-component is not in maintenance_state .

We can try following things here to get the correct status in UI

1) try restarting ambari-server if doesnt work Take a backup of database and look out for output of following commands

2) go to database, inspect the output of the following command :

ambari=> select id,component_name,desired_State,host_id,service_name,admin_state  from hostcomponentdesiredstate where service_name='HBASE';
 id  |   component_name   | desired_state | host_id | service_name | admin_state
-----+--------------------+---------------+---------+--------------+-------------
   9 | HBASE_MASTER       | STARTED       |       6 | HBASE        |
  27 | HBASE_REGIONSERVER | STARTED       |       2 | HBASE        | INSERVICE
  28 | HBASE_REGIONSERVER | STARTED       |       5 | HBASE        | INSERVICE
  86 | HBASE_CLIENT       | INSTALLED     |       1 | HBASE        |
  53 | HBASE_CLIENT       | INSTALLED     |       2 | HBASE        |
  54 | HBASE_CLIENT       | INSTALLED     |       5 | HBASE        |
  26 | HBASE_REGIONSERVER | STARTED       |       1 | HBASE        |
 151 | HBASE_CLIENT       | INSTALLED     |       6 | HBASE        |
 152 | HBASE_REGIONSERVER | STARTED       |       6 | HBASE        | INSERVICE
(9 rows)

3) correct the admin_state column if required. please remember to set it as NULL , example command :

ambari=> update hostcomponentdesiredstate set admin_state=NULL where id=<ID>;

4) restart ambari to see if there is any luck.

Note : Please take proper backup of the database and be 1000% sure of what command you are executing, make sure ambari-server is stopped before performing any edit to the database.If it's a production cluster always contact hortonworks support .

Hope this helps you troubleshooting more on this regard.

Please accept the answer if it helped.

avatar
Explorer

It displays started and a green check mark for a few seconds then eventually goes to a decommissioning status. It's as if there's a process that's checking something and changing the status

avatar
Explorer

I changed the admin_state to NULL and restarted ambari. The regionserver status still shows 'Decommissioning'. The rest api still shows desired_admin_state" : "INSERVICE.

avatar
Explorer
id | component_name | desired_state | host_id | service_name | admin_state 
-----+----------------------+---------------+---------+--------------+------------- 
154 | PHOENIX_QUERY_SERVER | STARTED | 1 | HBASE | INSERVICE 
155 | PHOENIX_QUERY_SERVER | STARTED | 2 | HBASE | INSERVICE 
159 | HBASE_CLIENT | INSTALLED | 2 | HBASE | 
157 | HBASE_CLIENT | INSTALLED | 1 | HBASE | 
158 | HBASE_CLIENT | INSTALLED | 3 | HBASE | 
156 | PHOENIX_QUERY_SERVER | STARTED | 3 | HBASE | INSERVICE 
152 | HBASE_REGIONSERVER | STARTED | 3 | HBASE | 
153 | HBASE_REGIONSERVER | STARTED | 1 | HBASE | 
151 | HBASE_MASTER | STARTED | 2 | HBASE |

  "href" : "https://hadoop-1.nit.disa.mil:8443/api/v1/clusters/PIEE/hosts/hadoop-2.nit.disa.mil/host_components/HBASE_REGIONSERVER/?fields=*",
  "HostRoles" : {
    "cluster_name" : "PIEE",
    "component_name" : "HBASE_REGIONSERVER",
    "desired_admin_state" : "INSERVICE",
    "desired_repository_version" : "3.0.1.0-187",
    "desired_stack_id" : "HDP-3.0",
    "desired_state" : "STARTED",
    "display_name" : "RegionServer",
    "host_name" : "hadoop-2.nit.disa.mil",
    "maintenance_state" : "OFF",
    "public_host_name" : "hadoop-2.nit.disa.mil",
    "reload_configs" : false,
    "service_name" : "HBASE",
    "stale_configs" : false,
    "state" : "STARTED",
    "upgrade_state" : "NONE",
    "version" : "3.0.1.0-187",
    "actual_configs" : { }
  }

avatar
Explorer

Hi,

I updated the hostcomponentdesiredstate table but the regionserver component is still showing 'Decommissioning'.

ambari=> select id,component_name,desired_State,host_id,service_name,admin_state  from hostcomponentdesiredstate where service_name='HBASE';
 id  |    component_name    | desired_state | host_id | service_name | admin_state
-----+----------------------+---------------+---------+--------------+-------------
 154 | PHOENIX_QUERY_SERVER | STARTED       |       1 | HBASE        | INSERVICE
 155 | PHOENIX_QUERY_SERVER | STARTED       |       2 | HBASE        | INSERVICE
 159 | HBASE_CLIENT         | INSTALLED     |       2 | HBASE        |
 157 | HBASE_CLIENT         | INSTALLED     |       1 | HBASE        |
 158 | HBASE_CLIENT         | INSTALLED     |       3 | HBASE        |
 156 | PHOENIX_QUERY_SERVER | STARTED       |       3 | HBASE        | INSERVICE
 152 | HBASE_REGIONSERVER   | STARTED       |       3 | HBASE        |
 153 | HBASE_REGIONSERVER   | STARTED       |       1 | HBASE        |
 151 | HBASE_MASTER         | STARTED       |       2 | HBASE        |
(9 rows)


The desired_admin_state still says 'inservice' in the rest api.

 "HostRoles" : {
    "cluster_name" : "PIEE",
    "component_name" : "HBASE_REGIONSERVER",
    "desired_admin_state" : "INSERVICE",
    "desired_repository_version" : "3.0.1.0-187",
    "desired_stack_id" : "HDP-3.0",
    "desired_state" : "STARTED",
    "display_name" : "RegionServer",
    "host_name" : "hadoop-2.nit.disa.mil",
    "maintenance_state" : "OFF",
    "public_host_name" : "hadoop-2.nit.disa.mil",
    "reload_configs" : false,
    "service_name" : "HBASE",
    "stale_configs" : false,
    "state" : "STARTED",
    "upgrade_state" : "NONE",
    "version" : "3.0.1.0-187",
    "actual_configs" : { }
  },

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login