Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HBase Cluster

HBase Cluster

Hello All,

 

In my cluster enviroment Master service is shutting down automatically.Here are logs

 

2014-07-17 09:09:47,232 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown.
org.apache.hadoop.hdfs.BlockMissingException: Could not obtain block: BP-1967360323-10.0.0.247-1404994306462:blk_-658212430220773160_1004 file=/hbase/hbase.id

 

Any hints ?

1 REPLY 1
Highlighted

Re: HBase Cluster

Master Guru
Somehow your file of /hbase/hbase.id has lost its only block, turning corrupt. I'd recommend inspecting your HDFS health (via 'hdfs fsck /', and also by checking if all DNs are alive and in good health).

In terms of recovery, if you are unable to recover the block back from the DataNodes, then you can proceed with stopping all HBase services, deleting the /hbase/hbase.id file, and then restarting the HBase service.

P.s. Another easier fix would be to use HBCK, but the required fixing feature (for cluster IDs) is incomplete yet: https://issues.apache.org/jira/browse/HBASE-7191