Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

hbase Master failed to become active master


hbase Master failed to become active master


My primary master died due to scheduled maintenance and my backup master failed to kick in 

CM Agent has tried to start it up but could not initialize the namespace table 

After several manual efforts where I followed


I did the following only 

rmr /hbase/meta-region-server
rmr /hbase/rs
rmr /hbase/splitWAL
rmr /hbase/backup-masters
rmr /hbase/table-lock
rmr /hbase/flush-table-proc
rmr /hbase/region-in-transition
rmr /hbase/running
rmr /hbase/balancer 
rmr /hbase/recovering-regions
 rmr /hbase/draining
 rmr /hbase/namespace 
 rmr /hbase/hbaseid
 rmr /hbase/table

I got the a master to come up after setting hbase.master.namespace.init.timeout to some absurd value

I see the master registering dead region servers (though I cannot find where it pick them up, not in the WAL, Archive or data)

and I see the master registering the following 

Starting namespace manager (since 1hrs, 20mins, 5sec ago)

even though cloudera manager shows healthy


list the catalog in hbase shell gives me the following error


hbase(main):004:0> list

ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
    at org.apache.hadoop.hbase.master.HMaster.checkInitialized(
    at org.apache.hadoop.hbase.master.MasterRpcServices.getTableNames(
    at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(
    at org.apache.hadoop.hbase.ipc.RpcExecutor$
    at org.apache.hadoop.hbase.ipc.RpcExecutor$

fsck /hbase -files -blocks shows healthy

hbck  shows zero inconsistencies


I am on version 

Hadoop 2.6.0+cdh5.11.1+2400
HBase 1.2.0+cdh5.11.1+319

I did have a master colocated with a region server and was wondering if I ran into this

and then this

as the cause of the failed backup kicking in 


But I cannot determine why I am getting namespace manager would take so long to initialize



Re: hbase Master failed to become active master


add on 

and master is still starting 

Starting namespace manager (since 1hrs, 55mins, 57sec ago)


Re: hbase Master failed to become active master


Another update

left master alone to see if it can resolve

Starting namespace manager (since 11hrs, 55mins, 3sec ago)

unfortunately I cannot find any logs on what exactly its trying to do



Re: hbase Master failed to become active master


Wish we could have fixed this for you... The only ideas I had involved some serious low level debugging / tracing while it was stuck in this state. Hope things are better now. 


Re: hbase Master failed to become active master

Cloudera Employee

Not sure which version of CDH you met with this issue. Note that the hbase namespace table is a system table required for HBase to function properly.  This table is not handled with a higher priority than other tables as noted in HBASE-14190. If the HBase Master tries to split a WALs on dead or ghost Region Servers then the HBase Master might get stuck trying to split these WALs on invalid Region Servers. The HBase Master can also get stuck trying to split corrupt 83-byte or smaller WAL files on startup in which case just sidelining those blocking WAL files would help. So increasing the hbase.master.namespace.init.timeout may or may not help depending on whether the Master is stuck with any other tasks like WAL splitting.


Another workaround is to recreate the hbase:namespace table in which case it will get loaded quickly.

Don't have an account?
Coming from Hortonworks? Activate your account here