Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HBASE Inconsistencies

Highlighted

HBASE Inconsistencies

Explorer

Hi Team ,

i ran hbase hbck command today and found many inconsistencies,please help how to fix those inconsistencies.

Hbase version =1.1.1.2.3

Thanks.

75 inconsistencies detected.
Status: INCONSISTENT

13 REPLIES 13
Highlighted

Re: HBASE Inconsistencies

Explorer

Could you post the details of inconsistencies?

Highlighted

Re: HBASE Inconsistencies

Explorer

Hi ,

ERROR: Region { meta => prod:testj,e147adec,1486082394593.ab7e50343530f345972a9ff03d4a0e22., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,51eb8510,1486082394593.ad388fcfa27cd267312411ade9d00df4., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,8a3d708b,1486082394593.ada4f1493f2fa6efcf7f900f8ecf44f0., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,4cccccbf,1486082394593.ae21fbc4a620299bf08f108d50d18a7d., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,5c28f5b2,1486082394593.b5b2a14adbd92cf79d014e0024c40e39., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,d70a3d4a,1486082394593.b82d3360761b7e92db29ab7f10e9b93b., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,147ae144,1486082394593.bd564c1236207ad260de7fbee8d263b0., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,0f5c28f3,1486082394593.d20d0ee18f4b351a71ae310d601e11e0., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,8f5c28dc,1486082394593.d694e5124817e5273915590b2ad621c7., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,eb851e8e,1486082394593.da8105162f9d0a0d28b0f1720dae1dbf., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,2e147ad9,1486082394593.e21fc0c48e61496d760d12e2dc741641., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,47ae146e,1486082394593.e245fd5c5954fb11a426f8b632fe4f00., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,fae14781,1486082394593.e9e56c74a1a2231664424d128e53d27f., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,c7ae1457,1486082394593.eeb813c6c9645c6b88d36ab10281656e., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,,1486082394593.f351780e45185669b6a89dac75b0616c., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,9999997e,1486082394593.f439f910a64cdc62879b03567d069894., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,3851eb7b,1486082394593.fbae2d87dae40da1abf376c03dd2ecbd., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.

Many Thanks.

Highlighted

Re: HBASE Inconsistencies

Can you attach HBase Master log and Region In Transition section from HBase Master UI screen?

Re: HBASE Inconsistencies

Explorer

Hi ,

Please find regions in transition screen,and Master log is too big to upload.

Thanks.12412-regions-in-transition.png

Highlighted

Re: HBASE Inconsistencies

Try following:

1. Assign region manually from hbase shell and monitor UI

#hbase> assign 'b4cc...'

2. Run following multiple times:

hbase hbck -fixAssignments -fixMeta -fixReferenceFiles

let me know the outcome.

Highlighted

Re: HBASE Inconsistencies

Guru

You should run HBCK with -fix argument(s).

Highlighted

Re: HBASE Inconsistencies

Explorer

What this means that the meta data of the region is in hbase system table (hbase:meta), but we could not find the data directories for the region in the filesystem (HDFS). One possibility is that somehow someone manually deleted the data directories. You could manually check whether the data is indeed does not exist.

Or you can repair it using the hbase hbck -repair' tool (which will create the directories to make it consistent). You might need to run a couple of times to finally fix the issue.

Note: one more thing, my suggestion is based on all your 75 inconsistencies are the same type you pasted.

Highlighted

Re: HBASE Inconsistencies

Explorer

@sjiang

i am planning to execute below command (table name=corrupted table name) ,does the below command going to solve the inconsistencies.

Please suggest.

Many Thanks.

hbase hbck Tablename
Highlighted

Re: HBASE Inconsistencies

Explorer

no, this only checks inconsistency, you need to add '-repair' argument

Don't have an account?
Coming from Hortonworks? Activate your account here