Hbase - How to fix failed regions

WayneWang — Fri, 16 Sep 2022 14:40:24 GMT

Hi,

we have a 3 datanode hadoop cluster (CDH 5.10). the replication factor is set to 2. now we cannot access data from Hbase table due to failed regions on that table. could somebody can give me some suggestion to fix this issue(we don't lost data)?

the failed regions are in "Regions in Transition" state. we tried to fix and repair the table by below command.

$ hbase hbck -fix <table> $ hbase hbck -repair <table>

But failed, we got error like below:

21/02/08 02:03:33 INFO util.HBaseFsck: Checking and fixing region consistency ERROR: Region { meta => xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952., hdfs => hdfs://cdh-cm.test.com:8020/hbase/data/default/xxxxLogsIndexer/327a6ed6b00c95e63346f6d725147952, deployed => , replicaId => 0 } not deployed on any region server. Trying to fix unassigned region... 21/02/08 02:03:33 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'} 21/02/08 02:03:34 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'} 21/02/08 02:03:35 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'} 21/02/08 02:03:36 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'} 21/02/08 02:03:37 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'} 21/02/08 02:03:38 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'} ^C21/02/08 02:03:39 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService 21/02/08 02:03:39 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x1777c1d848d09be 21/02/08 02:03:39 INFO zookeeper.ZooKeeper: Session: 0x1777c1d848d09be closed 21/02/08 02:03:39 INFO zookeeper.ClientCnxn: EventThread shut down 21/02/08 02:03:39 INFO util.HBaseFsck: Finishing hbck

Thanks!

Br, Wayne

Re: Hbase - How to fix failed regions

smdas — Tue, 02 Mar 2021 05:26:37 GMT

Hello @WayneWang

Thanks for using Cloudera Community. The issue being faced by your Team is a Table having 1 "Failed Region" i.e. 327a6ed6b00c95e63346f6d725147952. Your Team have tried HBCK fix & repair (As this is HBase v1.x), yet the issue persists.

We can try the following Steps as well:

Restarting the Masters after running any HBCK fix Command
As RIT are maintained in ZNode, Try removing the RIT ZNode & restart the Master again.
For the Region, Would request you to review the RegionServer (Wherein the Region is in FAILED State) & confirm the reasoning for the same as well.

- Smarak

Re: Hbase - How to fix failed regions

smdas — Sun, 14 Mar 2021 07:59:55 GMT

Hello @WayneWang

Hope you are doing well. I wish to follow-up with you concerning the issue posted. Kindly let us know if the concerned issue has been resolved & the steps followed. This would help us to proceed accordingly on the Post.

- Smarak

Re: Hbase - How to fix failed regions

smdas — Fri, 19 Mar 2021 07:12:35 GMT

Hello @WayneWang

As we haven't received any further Update, We are closing the Post assuming the issue was handled by the Steps shared above [1]. In HBase v1.x, We have limited choices with HBase v2.x using a new AssignmentManager (Details in HBASE-12439), which would assist in managing the RIT without ZooKeeper involvement.

Thanks for using Cloudera Community.

- Smarak

[1] https://community.cloudera.com/t5/Support-Questions/Hbase-How-to-fix-failed-regions/m-p/312263/highlight/true#M225066

question Re: Hbase - How to fix failed regions in Support Questions

Hbase - How to fix failed regions

Re: Hbase - How to fix failed regions

Re: Hbase - How to fix failed regions

Re: Hbase - How to fix failed regions