Hbase - How to fix failed regions

[#1]

Hi, 

we have a 3 datanode hadoop cluster (CDH 5.10). the replication factor is set to 2.  now we cannot access data from Hbase table due to failed regions on that table. could somebody can give me some suggestion to fix this issue(we don't lost data)? 

the failed regions are in "Regions in Transition" state. we tried to fix and repair the table by below command.

$ hbase hbck -fix <table>
$ hbase hbck -repair <table>

But failed, we got error like below:

21/02/08 02:03:33 INFO util.HBaseFsck: Checking and fixing region consistency
ERROR: Region { meta => xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952., hdfs => hdfs://cdh-cm.test.com:8020/hbase/data/default/xxxxLogsIndexer/327a6ed6b00c95e63346f6d725147952, deployed => , replicaId => 0 } not deployed on any region server.
Trying to fix unassigned region...
21/02/08 02:03:33 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'}
21/02/08 02:03:34 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'}
21/02/08 02:03:35 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'}
21/02/08 02:03:36 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'}
21/02/08 02:03:37 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'}
21/02/08 02:03:38 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'}
^C21/02/08 02:03:39 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
21/02/08 02:03:39 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x1777c1d848d09be
21/02/08 02:03:39 INFO zookeeper.ZooKeeper: Session: 0x1777c1d848d09be closed
21/02/08 02:03:39 INFO zookeeper.ClientCnxn: EventThread shut down
21/02/08 02:03:39 INFO util.HBaseFsck: Finishing hbck

Thanks!

Br, Wayne

[#2]

Hello @WayneWang 

Thanks for using Cloudera Community. The issue being faced by your Team is a Table having 1 "Failed Region" i.e. 327a6ed6b00c95e63346f6d725147952. Your Team have tried HBCK fix & repair (As this is HBase v1.x), yet the issue persists. 

We can try the following Steps as well:

• Restarting the Masters after running any HBCK fix Command
• As RIT are maintained in ZNode, Try removing the RIT ZNode & restart the Master again. 
• For the Region, Would request you to review the RegionServer (Wherein the Region is in FAILED State) & confirm the reasoning for the same as well. 

- Smarak

[#3]

Hello @WayneWang 

Hope you are doing well. I wish to follow-up with you concerning the issue posted. Kindly let us know if the concerned issue has been resolved & the steps followed. This would help us to proceed accordingly on the Post.

- Smarak

[#4]

Hello @WayneWang 

As we haven't received any further Update, We are closing the Post assuming the issue was handled by the Steps shared above [1]. In HBase v1.x, We have limited choices with HBase v2.x using a new AssignmentManager (Details in HBASE-12439), which would assist in managing the RIT without ZooKeeper involvement.

Thanks for using Cloudera Community.

- Smarak

[1] https://community.cloudera.com/t5/Support-Questions/Hbase-How-to-fix-failed-regions/m-p/312263/highl...