Hello @ASIF123    Checking whether the concerned issue posted by your team has been resolved. If Yes, Kindly mark the Post as Solved. If No, Kindly review our Post dated 12/18 & share the outcome of the Action Plan shared.   - Smarak ... View more

Hello @TGH    Yes, After doing any HBCK2 Changes, Restart the Service as the Components have a Cached Version of the Metadata as well. Let us know how things goes.    - Smarak ... View more

Hello @Anks2411    Thanks for using Cloudera Community. You shall need to check the Logs of " cdh-dn-28.prod.mcs.az-eastus2.mob.nuance.com" RegionServer to confirm the reasoning for Region "8808c0e1917bf0b4acea2d83d9548463" being FAILED_CLOSE.   For any Region to be moved from RegionServer A to RegionServer B, the Region has been to closed on RegionServer A before being opened on RegionServer B. In your case, the Region is failing to close on RegionServer A & the Logs would confirm the reasoning for the same.    - Smarak ... View more

@TGH    No worries, if you are planning to drop the Table anyhow, Let's use the following approaches: 1. You (Your previous Team) have deleted the Table Level Directories from HDFS.  2. In HBase:Meta, We have 1 Row per Table Region & 1 Row for Table as well.  3. Use "get 'hbase:meta','<RegionID RowKey>'". Note that your Team can get the Scan Output to check the format of the RowKey for the concerned Table, which includes the RegionID.  4. After confirming the Output from "get" Command, Use the "deleteall" Command with the same Argument to remove the Rows of Table's Region. Finally, Remove the Table Level Info as well.  5. Restart the HBase Service to clear the Cache as well.    Recommend to test the above on 1 Sample Table (Create Table > Remove Region Directory > Remove Meta Info > Restart HBase > Check HBCK Output).    - Smarak ... View more

Hello @TGH    I downloaded the HBCK2 Tool from the Steps shared & I could see the " extraRegionsInMeta" listed in the "README.md" file.   - Smarak ... View more

Hello @kvinod    As Cluster Replication wasn't being used based on the fact that "list_peers" isn't showing any Peer, It's likely the CleanerChore Thread wasn't performing its duties. Note that WALs are moved to oldWALs once the Last SequenceIDs of the WALs have been persisted to Disk via MemStore Flush. In other words, oldWALs being present doesn't necessarily means that the WALs are being persisted for replication. Now, the Cleanup of oldWALs is CleanerChore Thread responsibility. As we covered above, the HBase Service Restart covered the HMaster Restart, which would ensure the CleanerChore Thread is spawned afresh.    Let me know if the above answers your queries.   - Smarak ... View more

Hello @TGH    Sharing the Steps for building the HBCK2 Jar using Git reference & additionally, refer the Post via [1] for the details on building HBCK2 Tool as well.     - Smarak   [1] https://community.cloudera.com/t5/Support-Questions/How-to-get-hbck2-tool-for-CDH-6-3-2/m-p/295867/highlight/true#M218004 ... View more

Hello @TGH    Thanks for using Cloudera Community. You had Region-In-Transition (RIT) & the HDFS Directory has been removed for the Regions along with the ZNode being removed, yet HBase reports RIT. You wish to fix the RIT issue by removing the Meta Table entries as RIT avoids Balancer run. In HBase v2 (CDH v6.3.x), the MasterProcWALs is critical for any Procedure, which are stuck or blocked. You mentioned a lot of procedures (Disable|Delete) being observed.    The graceful manner for your Team to manage the requirement is to use the HBCK2 Tool. You can build the HBCK2 using the Link [1]. Next, You can use the HBCK2 Tool to bypass the Procedure (PIDs) associated with the Table, for which the Region Directories have been removed. Once any PID is bypassed, the HMaster UI Page (Locks & Procedures ) Section would show the PID as "Bypass". After ensuring the required PIDs are bypassed, Restart the HMaster Service & use the HBCK2 Tool to remove the Region entries in Meta, for which the HDFS Region Directories are removed. Use "bypass" & "extraRegionsInMeta" HBCK2 Command as documented in Link [1].   Alternatively, You can Stop HMaster > Remove MasterProcWALs (After confirming no RUNNABLE Procedures excluding the PID associated with the Table for which Region Directory have been removed) > Start HMaster. However, this isn't an ideal approach & you can encounter "Master Is Initialising" issue, for which HBCK2 Tool is required. The "Master Is Initialising" context is captured in Link [1] as well.   - Smarak   [1] https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2 ... View more