HBase HBCK2 tool for HDP 3.x

[#1]

In HDP 2.x we had hbck tool which was used to identify and fix inconsistencies in hbase tables. However with HBase 2.x and HDP 3.x together this tool is deprecated and its strongly advised to not run it on production clusters as it may lead to data corruption.

HBase 2.0 introduces a completely revamped HBase Assignment Manager with Proc V2 framework (State machine). In HDP-2.x, we maintained HBASE_REGION_STATE and TABLE_STATE in zookeeper and Hmaster in-memory ( possibility for inconsistencies) but with HDP-3 it uses MasterProcWAL procedure which is stored in HDFS.

These set of procedures are now scheduled and executed by HMaster.

Given the changes on internals of new AssignmentManager , the old hbck fix options no longer work, but it can still provide a report about tables/regions states. To fix regions assignment problems that may still occur, a new HBCK 2 has been designed, and shipped independently from hbase build artifacts (and hence not part of HDP 3.x distribution ), as a plan to have it able to evolve on its own, with new fix options for previously unforeseen issues added and integrated into the tool without the need of whole new hbase release.

This HBCK2 tool as of now is not as intuitive and useful as old hbck but it helps fix issues related to Proc v2 (introduced in HBase 2.0) and other issues such as region in transition.

Note: HBCK2 tool requires minimum HBase 2.0.3 which comes starting HDP 3.1

To build hbase-hbck2:
# cd <work directory>
# git clone https://github.com/apache/hbase-operator-tools.git
# cd hbase-operator-tools/
# mvn clean install -DskipTests
# cd hbase-hbck2/target
# Download hbase-hbck2-1.0.0-SNAPSHOT.jar


Running HBCK2

Once HBCK2 has been uploaded to the target cluster, it can be executed by specifying its jar path in the "-j" option of "hbck" command as below:

# su - hbase
$ hbase hbck -j <jar path>/hbase-hbck2-1.0.0-SNAPSHOT.jar <COMMAD>

You may encounter following error running in HDP 3.0

Exception in thread "main" java.lang.UnsupportedOperationException: Requires 2.0.3 at least.

Use -s option, instead.
$ hbase hbck -j <jar path>/hbase-hbck2-1.0.0-SNAPSHOT.jar -s <COMMAND>

Reference:

HBASE-14350
HBASE-19121
HBase-Operator-Tool

Special mention and thanks to Karthik Palanisamy for critical inputs here.

[#2]

Hi

HDP 3.1 use hbase 2.0.2, but HBCK2 tool requires minimum HBase 2.0.3,How can I use it。

cluster ：HDP3.1.0 on Kerberos

error：