HBase Master Running but not Active

[#1]

Hi -

We are running an issue with HBase where the Master is running but not active.

When I go to HBase shell and run 'List' command I get below error :

hbase(main):001:0> list
TABLE

ERROR: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2371)
at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:938)
at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55654)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2191)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:183)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:163)

List all tables in hbase. Optional regular expression parameter could
be used to filter the output. Examples:

hbase> list
hbase> list 'abc.*'
hbase> list 'ns:abc.*'
hbase> list 'ns:.*'

We have configured the 'Master' role in Cloudera Manager. The service starts properly and shows 'green' icon which says the service started succesfully.

But when when we go to hbase master log this was one of the error it showed :

ERROR org.apache.hadoop.hbase.master.HMaster: Master failed to complete initialization after 900000ms. Please consider submitting a bug report including a thread dump of this process.

Moreover, the HBase service page which shows all the instances list, shows Master instance in green color. But it does not shows Master (Active) state.

[#2]

Hi,

More update on this :

We upgraded our Cloudera Manager & CDH Version to 6.2 after following all the required steps.

Now, we can see that Master is active and Backup Master is also running and identified and assumed responsibility.

Problem :

When we go to hbase shell & run 'list' command we see all the tables that should be there.

When we run 'count' or 'scan' command we get below error in hbase shell :

"ERROR: No server address listed in hbase:meta for region volume,,1547112088868.09539931eb256ed2dfb5a43c116e0112. containing row"

Below is the error on HBase Master Log :

ABORTING master HOST,16000,1558270209713: Unhandled exception. Starting shutdown. *****
java.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILED
at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:345)
at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:291)
at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1260)
at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1038)
at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2271)
at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:581)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Timedout 300000ms waiting for namespace table to be assigned and enabled: tableName=hbase:namespace, state=ENABLED
at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:108)
at org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:63)
at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:226)
at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1258)
... 4 more