hbase Master startup cannot progress

[#1]

Hi Team,

we are getting below error org.apache.hadoop.hbase.master.HMaster: hbase:namespace,,1607693423707.761db88b0c48d79753d50f0f158eaece. is NOT online; state={761db88b0c48d79753d50f0f158eaece

when we are cheking the status of hbase (status command) we are getting same region server in live status as well in dead also.

We ran the below command to fix it 

hbase hbck -j hbase-hbck2-1.2.0-SNAPSHOT.jar -s assigns hbase:namespace 1607693423787.761db88b0c48d79753d50f0f158ejkl but the issue is still there.

the command is completing like below.

 org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient - Close zookeeper connection 0x0b86de0d to

Regards,

Satya 

[#2]

Hello @Satya_Singh 

Thanks for using Cloudera Community. Based on the Post, HBase Master isn't progressing on StartUp owing to "hbase:namespace" being not Online. Your Team tried to assign the Region via HBCK2 yet the issue persists. Additionally, the RegionServer on which the "hbase:namespace" Region is marked for assignment is under Live & Dead as well. 

The approach used by your Team is correct with respect to using HBCK2. When your Team run the HBCK2 assign command, It should return a PID before terminating. The Output would be like [10], assuming "10" is the PID. Review the HMaster UI for the concerned PID & review the State. 

When an RS has Live & Dead Status, Review the StartUp Code for the Live & Dead RS. For Example, the RS "hbase-test-rc-5.openstacklocal,16020,1469419333913" StartUp Code is "1469419333913". We expect the Live RS StartUp Code would be recent & Dead RS StartUp Code would be Older with the "hbase:namespace" Region being tried to be assigned to the Live RS. Use the Epoch Converter via Link [1] to convert the StartUp Code to Human Readable Timestamp. 

If the PID State is fine with Region being assigned to the RS with Live StartUp Code, Review the concerned RS Logs to confirm if the Region Assignment has been received & any issues with opening the Region. Accordingly, Proceed with the troubleshooting. 

Do review & let us know your observations. 

- Smarak

[1] https://www.epochconverter.com/

[#3]

Hello @Satya_Singh 

Do let us know if your issue has been resolved. If Yes, Please share the Mitigation Steps followed by you to ensure other Community Users can benefit from your experience & mark the Post as Resolved as well. 

- Smarak

[#4]

Hello @Satya_Singh 

Do let us know if your issue has been resolved. If Yes, Please share the Mitigation Steps followed by you to ensure other Community Users can benefit from your experience & mark the Post as Resolved as well. 

- Smarak