hbase I get an error "Master is initializing"

[#1]

we are having a problem with hbase, the following error is displayed:

INFO client.RpcRetryingCallerImpl: Call exception, tries=6, retries=11, started=4175 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2982)
at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:1018)
at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
, details=, see https://s.apache.org/timeout

hbase and zookeeper have already been restarted several times and still the above error is presented

the zookeeper "version-2" directory has already been removed and the above error is still displayed

how can we solve this problem?

[#2]

Hi @yagoaparecidoti ,

You need to check the full master logs to check the cause of the issue.

Check before "Master is initializing" part in the logs whether any system tables are stuck.

Generally when hbase:meta or hbase:namespace tables are not assigned, we get this error.

Regards,

Shubham

[#3]

hi @shubham_sharma, how are you?

I found a command on the internet that possibly solved this error, but some tables ended up being broken, so it was necessary to delete the tables and recreate them again.

[#4 · Accepted solution]

Hello @yagoaparecidoti 

Thanks for using Cloudera Community. Based on the Post, You encountered "Master Is Initializing" & ended up fixing the same using Command found on Internet. You ended up with few Tables in Broken State, requiring you to delete & recreate them.

For documenting the Post, the Issue observed is tracked via [1]. In the HMaster Logs, We should see HBase:Meta & HBase:Namespace Region, which aren't being assigned. Sample tracing shared via [1]. Once the same happens, Using HBCK2 Jar to assign the Region being reported as Unassigned is required. The Command to be used via HBCK2 Jar to assign the Region is shared via [1] as well. 

The HBCK2 Command has other Options, which (If performed without any oversight) may cause issues with HBase Table availability. The "Master Is Initializing" Error Fix as shared via [1] shouldn't cause any Table to be in Broken State & without knowing the explicit details into the Broken State, It's harder to confirm if Delete-Recreate was the ONLY Way Or, We could have restored the Table in other ways.

Having said that, the Post covers "Master Is Initializing" Error & the same has been addressed. As such, We shall mark the Post as Closed. Feel free to share your observation with the Tables in Broken State in a New Post, if you wish to engage for the Community observation & feedback. 

Regards, Smarak

[1] https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2#master-startup-cannot-progres...