Hello @TGH    I downloaded the HBCK2 Tool from the Steps shared & I could see the "extraRegionsInMeta" listed in the "README.md" file.   - Smarak ... View more

Hello @Manoj690    Thanks for contacting Cloudera Community. While taking a Full Backup, you are facing IOException while waiting on the Lock.    Kindly share the Output of Command "hbase backup history" along with "list_locks" from HBase Shell. The requested details would confirm the status of any running Backup & Locks placed on the Tables. Additionally, Share the HBase Version wherein you are using the required Backup Command.    - Smarak ... View more

Hello @ShamsN    Thanks for using Cloudera Community. You mentioned the HBase RegionServers are shutting down & mentioned an error, yet we don't any error message excluding the UI. Please share the RegionServer logs to confirm the cause of termination. Additionally, What is the Error you are receiving while listing Tables.   - Smarak ... View more

Hello @kvinod    As Cluster Replication wasn't being used based on the fact that "list_peers" isn't showing any Peer, It's likely the CleanerChore Thread wasn't performing its duties. Note that WALs are moved to oldWALs once the Last SequenceIDs of the WALs have been persisted to Disk via MemStore Flush. In other words, oldWALs being present doesn't necessarily means that the WALs are being persisted for replication. Now, the Cleanup of oldWALs is CleanerChore Thread responsibility. As we covered above, the HBase Service Restart covered the HMaster Restart, which would ensure the CleanerChore Thread is spawned afresh.    Let me know if the above answers your queries.   - Smarak ... View more

Hello @tuk    If the Post by Pabitra assisted you, Kindly mark the Post as Solution. If you utilised any other approach, Kindly share the details in the post as well.   Thanks, Smarak ... View more

Hello @TGH    Sharing the Steps for building the HBCK2 Jar using Git reference & additionally, refer the Post via [1] for the details on building HBCK2 Tool as well.     - Smarak   [1] https://community.cloudera.com/t5/Support-Questions/How-to-get-hbck2-tool-for-CDH-6-3-2/m-p/295867/highlight/true#M218004 ... View more

Hello @TGH    Thanks for using Cloudera Community. You had Region-In-Transition (RIT) & the HDFS Directory has been removed for the Regions along with the ZNode being removed, yet HBase reports RIT. You wish to fix the RIT issue by removing the Meta Table entries as RIT avoids Balancer run. In HBase v2 (CDH v6.3.x), the MasterProcWALs is critical for any Procedure, which are stuck or blocked. You mentioned a lot of procedures (Disable|Delete) being observed.    The graceful manner for your Team to manage the requirement is to use the HBCK2 Tool. You can build the HBCK2 using the Link [1]. Next, You can use the HBCK2 Tool to bypass the Procedure (PIDs) associated with the Table, for which the Region Directories have been removed. Once any PID is bypassed, the HMaster UI Page (Locks & Procedures ) Section would show the PID as "Bypass". After ensuring the required PIDs are bypassed, Restart the HMaster Service & use the HBCK2 Tool to remove the Region entries in Meta, for which the HDFS Region Directories are removed. Use "bypass" & "extraRegionsInMeta" HBCK2 Command as documented in Link [1].   Alternatively, You can Stop HMaster > Remove MasterProcWALs (After confirming no RUNNABLE Procedures excluding the PID associated with the Table for which Region Directory have been removed) > Start HMaster. However, this isn't an ideal approach & you can encounter "Master Is Initialising" issue, for which HBCK2 Tool is required. The "Master Is Initialising" context is captured in Link [1] as well.   - Smarak   [1] https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2 ... View more