Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

hbase hbck reports inconsistency immediately after adding hbase service

avatar
Explorer

Hi everyone!

 

So a reeeally long story short (I can gladly expand upon request) - I added the hbase service, did hbase hbck immediately after this and it already detected one inconsistency:

 

ERROR: Region { meta => hbase:namespace,,1485505125654.b972bf2653eaa96104d6034591386a60., 
hdfs => null, deployed => hadoop-34.xxxzzz.de,60020,1485505116059;hbase:namespace,,
1485505125654.b972bf2653eaa96104d6034591386a60., replicaId => 0 }
found in META, but not in HDFS, and deployed on hadoop-34.xxxzzz.de,60020,1485505116059

When I do hbase hbck -repairHoles the inconsistency is gone, BUT... so is my hbase:namespace table.

 

hbase(main):001:0> scan 'hbase:namespace'
ROW                                    COLUMN+CELL 

ERROR: Unknown table hbase:namespace!

Interestingly enough, not gone from HDFS:

 

hdfs dfs -ls /hbase/data/hbase
Found 2 items
drwxr-xr-x   - hbase hbase          0 2017-01-27 09:18 /hbase/data/hbase/meta
drwxr-xr-x   - hbase hbase          0 2017-01-27 09:18 /hbase/data/hbase/namespace

...nor from the the zookeeper:

 

[zk: localhost:2181(CONNECTED) 2] ls /hbase/table
[hbase:meta, hbase:namespace]

...and an interesting side effect is that create_namespace function of the hbase shell is now gone:

 

hbase(main):003:0> create_namespace 'ns1'

ERROR: Unknown table ns1!

 

I did find this ray of hope: HBASE-16294 and this is actually included in latest CDH (I am running 5.9.0 btw).

 

But!

 

This seems to concern only replicas. This is the patch code btw:

 

if (hbi.getReplicaId() == HRegionInfo.DEFAULT_REPLICA_ID) {
 // Log warning only for default/ primary replica with no region dir
 LOG.warn("No HDFS region dir found: " + hbi + " meta=" + hbi.metaEntry);
}

 

I have replication disabled, and as one can see from the error message: 

replicaId => 0

 

Now, I would have let this slide, but the real problem is that over time I get a huge number of these inconsistencies and attempt to fix them results in not being able to find tables from hbase shell.

 

Any ideas would be greatly appreciated!

1 ACCEPTED SOLUTION

avatar
Master Collaborator

crazy thought: does the node that you are running hbck on have the HDFS gateway role applied?

 

could be that hbck can't find the region in HDFS because it doesn't know how to connect to hdfs?

another way to verify would be to check the hdfs location for the hbase tables:

 

/hbase/data/default/<table>

View solution in original post

11 REPLIES 11

avatar
Rising Star

Hello Everyone,

 

I am seein the below error's in hbase when i checked the inconsistency of hbase.

 

ERROR: Found inconsistency in table SYSTEM.CATALOG 

ERROR: There is a hole in the region chain between  and .  You need to create a new .regioninfo and region dir in hdfs to plug the hole

 

 

Can some one please address the issue and suggest me with valuable inputs.

Thansk in advance.

 

Vinod

 

avatar
Contributor

Hello Vinod,

 

Please refer to the previously mentioned hbck guide[1] or review the Appendix C[2] which is referenced at the end of that documentation that further discusses usage of hbck.

 

Generally near every hbck troubleshooting is best begin with a single

$ sudo -u hbase hbase hbck -fixAssignments

As this will try to assign all regions that are not deployed at the time of the running. Holes can be present by many different reasons, the first step is reviewing if every region is assigned successfully, would the hole persist. Reassigning regions successfully usually eliminates the holes in the region chain.

 

It's also a good practice to see if the Apache HBase "thebook" has any information about the issue at hand. As CDH5.8+ uses HBase 1.2 it's best to check out the corresponding version of the Apache Documentation on HBase[3]. If you would have CDH6.0.x then it's best to review HBase 2.0's documentation of the same[4] which has hbck2.

 

[1] - Checking and Repiring HBase tables CDH5.15.x - https://www.cloudera.com/documentation/enterprise/5-15-x/topics/admin_hbase_hbck.html

[2] - Apache HBase documentation v1.2 / Appendix C - http://hbase.apache.org/1.2/book.html#hbck.in.depth

[3] - Apache HBase documentation v1.2 / HBase hbck - http://hbase.apache.org/1.2/book.html#hbck

[4] - Apache HBase documentation / HBase HBCK2 - http://hbase.apache.org/book.html#HBCK2