hbase hbck reports inconsistency immediately after adding hbase service

[#1]

Hi everyone!

So a reeeally long story short (I can gladly expand upon request) - I added the hbase service, did hbase hbck immediately after this and it already detected one inconsistency:

ERROR: Region { meta => hbase:namespace,,1485505125654.b972bf2653eaa96104d6034591386a60., 
hdfs => null, deployed => hadoop-34.xxxzzz.de,60020,1485505116059;hbase:namespace,,
1485505125654.b972bf2653eaa96104d6034591386a60., replicaId => 0 } 
found in META, but not in HDFS, and deployed on hadoop-34.xxxzzz.de,60020,1485505116059

When I do hbase hbck -repairHoles the inconsistency is gone, BUT... so is my hbase:namespace table.

hbase(main):001:0> scan 'hbase:namespace'
ROW                                    COLUMN+CELL 

ERROR: Unknown table hbase:namespace!

Interestingly enough, not gone from HDFS:

hdfs dfs -ls /hbase/data/hbase
Found 2 items
drwxr-xr-x   - hbase hbase          0 2017-01-27 09:18 /hbase/data/hbase/meta
drwxr-xr-x   - hbase hbase          0 2017-01-27 09:18 /hbase/data/hbase/namespace

...nor from the the zookeeper:

[zk: localhost:2181(CONNECTED) 2] ls /hbase/table
[hbase:meta, hbase:namespace]

...and an interesting side effect is that create_namespace function of the hbase shell is now gone:

hbase(main):003:0> create_namespace 'ns1'

ERROR: Unknown table ns1!

I did find this ray of hope: HBASE-16294 and this is actually included in latest CDH (I am running 5.9.0 btw).

But!

This seems to concern only replicas. This is the patch code btw:

if (hbi.getReplicaId() == HRegionInfo.DEFAULT_REPLICA_ID) {
 // Log warning only for default/ primary replica with no region dir
 LOG.warn("No HDFS region dir found: " + hbi + " meta=" + hbi.metaEntry);
}

I have replication disabled, and as one can see from the error message: 

replicaId => 0

Now, I would have let this slide, but the real problem is that over time I get a huge number of these inconsistencies and attempt to fix them results in not being able to find tables from hbase shell.

Any ideas would be greatly appreciated!

[#2]

Hello Everyone,

I am seein the below error's in hbase when i checked the inconsistency of hbase.

ERROR: Found inconsistency in table SYSTEM.CATALOG 

ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo and region dir in hdfs to plug the hole

Can some one please address the issue and suggest me with valuable inputs.

Thansk in advance.

Vinod

[#3]

Hello Vinod,

Please refer to the previously mentioned hbck guide[1] or review the Appendix C[2] which is referenced at the end of that documentation that further discusses usage of hbck.

Generally near every hbck troubleshooting is best begin with a single

$ sudo -u hbase hbase hbck -fixAssignments

As this will try to assign all regions that are not deployed at the time of the running. Holes can be present by many different reasons, the first step is reviewing if every region is assigned successfully, would the hole persist. Reassigning regions successfully usually eliminates the holes in the region chain.

It's also a good practice to see if the Apache HBase "thebook" has any information about the issue at hand. As CDH5.8+ uses HBase 1.2 it's best to check out the corresponding version of the Apache Documentation on HBase[3]. If you would have CDH6.0.x then it's best to review HBase 2.0's documentation of the same[4] which has hbck2.

[1] - Checking and Repiring HBase tables CDH5.15.x - https://www.cloudera.com/documentation/enterprise/5-15-x/topics/admin_hbase_hbck.html

[2] - Apache HBase documentation v1.2 / Appendix C - http://hbase.apache.org/1.2/book.html#hbck.in.depth

[3] - Apache HBase documentation v1.2 / HBase hbck - http://hbase.apache.org/1.2/book.html#hbck

[4] - Apache HBase documentation / HBase HBCK2 - http://hbase.apache.org/book.html#HBCK2