Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HBase Master Running but not Active

HBase Master Running but not Active

New Contributor

Hi -

 

We are running an issue with HBase where the Master is running but not active.

 

When I go to HBase shell and run 'List' command I get below error :

 

hbase(main):001:0> list
TABLE

ERROR: org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet
at org.apache.hadoop.hbase.master.HMaster.checkServiceStarted(HMaster.java:2371)
at org.apache.hadoop.hbase.master.MasterRpcServices.isMasterRunning(MasterRpcServices.java:938)
at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:55654)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2191)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:183)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:163)

List all tables in hbase. Optional regular expression parameter could
be used to filter the output. Examples:

hbase> list
hbase> list 'abc.*'
hbase> list 'ns:abc.*'
hbase> list 'ns:.*'

 

We have configured the 'Master' role in Cloudera Manager. The service starts properly and shows 'green' icon which says the service started succesfully.

 

But when when we go to hbase master log this was one of the error it showed :

 

ERROR org.apache.hadoop.hbase.master.HMaster: Master failed to complete initialization after 900000ms. Please consider submitting a bug report including a thread dump of this process.

 

Moreover, the HBase service page which shows all the instances list, shows Master instance in green color. But it does not shows Master (Active) state.

1 REPLY 1

Re: HBase Master Running but not Active

New Contributor

Hi,

 

More update on this :

 

We upgraded our Cloudera Manager & CDH Version to 6.2 after following all the required steps.

 

Now, we can see that Master is active and Backup Master is also running and identified and assumed responsibility.

 

Problem :

 

When we go to hbase shell & run 'list' command we see all the tables that should be there.

 

When we run 'count' or 'scan' command we get below error in hbase shell :

 

"ERROR: No server address listed in hbase:meta for region volume,,1547112088868.09539931eb256ed2dfb5a43c116e0112. containing row"

 

 

Below is the error on HBase Master Log :

 

ABORTING master HOST,16000,1558270209713: Unhandled exception. Starting shutdown. *****
java.lang.IllegalStateException: Expected the service ClusterSchemaServiceImpl [FAILED] to be RUNNING, but the service has FAILED
at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.checkCurrentState(AbstractService.java:345)
at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.awaitRunning(AbstractService.java:291)
at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1260)
at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1038)
at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2271)
at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:581)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Timedout 300000ms waiting for namespace table to be assigned and enabled: tableName=hbase:namespace, state=ENABLED
at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:108)
at org.apache.hadoop.hbase.master.ClusterSchemaServiceImpl.doStart(ClusterSchemaServiceImpl.java:63)
at org.apache.hbase.thirdparty.com.google.common.util.concurrent.AbstractService.startAsync(AbstractService.java:226)
at org.apache.hadoop.hbase.master.HMaster.initClusterSchemaService(HMaster.java:1258)
... 4 more