HBASE Inconsistencies

[#1]

Hi Team ,

i ran hbase hbck command today and found many inconsistencies,please help how to fix those inconsistencies.

Hbase version =1.1.1.2.3

Thanks.

75 inconsistencies detected.
Status: INCONSISTENT

[#2]

Could you post the details of inconsistencies?

[#3]

Hi ,

ERROR: Region { meta => prod:testj,e147adec,1486082394593.ab7e50343530f345972a9ff03d4a0e22., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,51eb8510,1486082394593.ad388fcfa27cd267312411ade9d00df4., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,8a3d708b,1486082394593.ada4f1493f2fa6efcf7f900f8ecf44f0., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,4cccccbf,1486082394593.ae21fbc4a620299bf08f108d50d18a7d., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,5c28f5b2,1486082394593.b5b2a14adbd92cf79d014e0024c40e39., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,d70a3d4a,1486082394593.b82d3360761b7e92db29ab7f10e9b93b., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,147ae144,1486082394593.bd564c1236207ad260de7fbee8d263b0., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,0f5c28f3,1486082394593.d20d0ee18f4b351a71ae310d601e11e0., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,8f5c28dc,1486082394593.d694e5124817e5273915590b2ad621c7., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,eb851e8e,1486082394593.da8105162f9d0a0d28b0f1720dae1dbf., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,2e147ad9,1486082394593.e21fc0c48e61496d760d12e2dc741641., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,47ae146e,1486082394593.e245fd5c5954fb11a426f8b632fe4f00., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,fae14781,1486082394593.e9e56c74a1a2231664424d128e53d27f., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,c7ae1457,1486082394593.eeb813c6c9645c6b88d36ab10281656e., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,,1486082394593.f351780e45185669b6a89dac75b0616c., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,9999997e,1486082394593.f439f910a64cdc62879b03567d069894., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.
ERROR: Region { meta => prod:testj,3851eb7b,1486082394593.fbae2d87dae40da1abf376c03dd2ecbd., hdfs => null, deployed => , replicaId => 0 } found in META, but not in HDFS or deployed on any region server.

Many Thanks.

[#4]

Can you attach HBase Master log and Region In Transition section from HBase Master UI screen?

[#5]

Hi ,

Please find regions in transition screen,and Master log is too big to upload.

Thanks.

[#6]

Try following:

1. Assign region manually from hbase shell and monitor UI

#hbase> assign 'b4cc...'

2. Run following multiple times:

hbase hbck -fixAssignments -fixMeta -fixReferenceFiles

let me know the outcome.

[#7]

You should run HBCK with -fix argument(s).

[#8]

What this means that the meta data of the region is in hbase system table (hbase:meta), but we could not find the data directories for the region in the filesystem (HDFS). One possibility is that somehow someone manually deleted the data directories. You could manually check whether the data is indeed does not exist.

Or you can repair it using the hbase hbck -repair' tool (which will create the directories to make it consistent). You might need to run a couple of times to finally fix the issue.

Note: one more thing, my suggestion is based on all your 75 inconsistencies are the same type you pasted.

[#9]

@sjiang

i am planning to execute below command (table name=corrupted table name) ,does the below command going to solve the inconsistencies.

Please suggest.

Many Thanks.

hbase hbck Tablename

[#10]

no, this only checks inconsistency, you need to add '-repair' argument