Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HBASE Inconsistencies

HBASE Inconsistencies

New Contributor

Hi Team ,

i ran hbase hbck command today and found many inconsistencies,please help how to fix those inconsistencies.

Hbase version =1.1.1.2.3

Thanks.

75 inconsistencies detected.
Status: INCONSISTENT

13 REPLIES 13

Re: HBASE Inconsistencies

New Contributor

Could you post the details of inconsistencies?

Re: HBASE Inconsistencies

New Contributor

Hi ,

ERROR: Region { meta => prod:testj,e147adec,1486082394593.ab7e50343530f345972a9ff03d4a0e22., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,51eb8510,1486082394593.ad388fcfa27cd267312411ade9d00df4., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,8a3d708b,1486082394593.ada4f1493f2fa6efcf7f900f8ecf44f0., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,4cccccbf,1486082394593.ae21fbc4a620299bf08f108d50d18a7d., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,5c28f5b2,1486082394593.b5b2a14adbd92cf79d014e0024c40e39., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,d70a3d4a,1486082394593.b82d3360761b7e92db29ab7f10e9b93b., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,147ae144,1486082394593.bd564c1236207ad260de7fbee8d263b0., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,0f5c28f3,1486082394593.d20d0ee18f4b351a71ae310d601e11e0., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,8f5c28dc,1486082394593.d694e5124817e5273915590b2ad621c7., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,eb851e8e,1486082394593.da8105162f9d0a0d28b0f1720dae1dbf., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,2e147ad9,1486082394593.e21fc0c48e61496d760d12e2dc741641., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,47ae146e,1486082394593.e245fd5c5954fb11a426f8b632fe4f00., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,fae14781,1486082394593.e9e56c74a1a2231664424d128e53d27f., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,c7ae1457,1486082394593.eeb813c6c9645c6b88d36ab10281656e., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,,1486082394593.f351780e45185669b6a89dac75b0616c., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,9999997e,1486082394593.f439f910a64cdc62879b03567d069894., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,3851eb7b,1486082394593.fbae2d87dae40da1abf376c03dd2ecbd., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.

Many Thanks.

Re: HBASE Inconsistencies

Can you attach HBase Master log and Region In Transition section from HBase Master UI screen?

Re: HBASE Inconsistencies

New Contributor

Hi ,

Please find regions in transition screen,and Master log is too big to upload.

Thanks.12412-regions-in-transition.png

Re: HBASE Inconsistencies

Try following:

1. Assign region manually from hbase shell and monitor UI

#hbase> assign 'b4cc...'

2. Run following multiple times:

hbase hbck -fixAssignments -fixMeta -fixReferenceFiles

let me know the outcome.

Highlighted

Re: HBASE Inconsistencies

Guru

You should run HBCK with -fix argument(s).

Re: HBASE Inconsistencies

New Contributor

What this means that the meta data of the region is in hbase system table (hbase:meta), but we could not find the data directories for the region in the filesystem (HDFS). One possibility is that somehow someone manually deleted the data directories. You could manually check whether the data is indeed does not exist.

Or you can repair it using the hbase hbck -repair' tool (which will create the directories to make it consistent). You might need to run a couple of times to finally fix the issue.

Note: one more thing, my suggestion is based on all your 75 inconsistencies are the same type you pasted.

Re: HBASE Inconsistencies

New Contributor

@sjiang

i am planning to execute below command (table name=corrupted table name) ,does the below command going to solve the inconsistencies.

Please suggest.

Many Thanks.

hbase hbck Tablename

Re: HBASE Inconsistencies

New Contributor

no, this only checks inconsistency, you need to add '-repair' argument

Don't have an account?
Coming from Hortonworks? Activate your account here