Created on 02-07-2021 06:24 PM - edited 09-16-2022 07:40 AM
Hi,
we have a 3 datanode hadoop cluster (CDH 5.10). the replication factor is set to 2. now we cannot access data from Hbase table due to failed regions on that table. could somebody can give me some suggestion to fix this issue(we don't lost data)?
the failed regions are in "Regions in Transition" state. we tried to fix and repair the table by below command.
$ hbase hbck -fix <table>
$ hbase hbck -repair <table>
But failed, we got error like below:
21/02/08 02:03:33 INFO util.HBaseFsck: Checking and fixing region consistency
ERROR: Region { meta => xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952., hdfs => hdfs://cdh-cm.test.com:8020/hbase/data/default/xxxxLogsIndexer/327a6ed6b00c95e63346f6d725147952, deployed => , replicaId => 0 } not deployed on any region server.
Trying to fix unassigned region...
21/02/08 02:03:33 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'}
21/02/08 02:03:34 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'}
21/02/08 02:03:35 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'}
21/02/08 02:03:36 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'}
21/02/08 02:03:37 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'}
21/02/08 02:03:38 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'}
^C21/02/08 02:03:39 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
21/02/08 02:03:39 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x1777c1d848d09be
21/02/08 02:03:39 INFO zookeeper.ZooKeeper: Session: 0x1777c1d848d09be closed
21/02/08 02:03:39 INFO zookeeper.ClientCnxn: EventThread shut down
21/02/08 02:03:39 INFO util.HBaseFsck: Finishing hbck
Thanks!
Br, Wayne
Created 03-01-2021 09:26 PM
Hello @WayneWang
Thanks for using Cloudera Community. The issue being faced by your Team is a Table having 1 "Failed Region" i.e. 327a6ed6b00c95e63346f6d725147952. Your Team have tried HBCK fix & repair (As this is HBase v1.x), yet the issue persists.
We can try the following Steps as well:
- Smarak
Created 03-13-2021 11:59 PM
Hello @WayneWang
Hope you are doing well. I wish to follow-up with you concerning the issue posted. Kindly let us know if the concerned issue has been resolved & the steps followed. This would help us to proceed accordingly on the Post.
- Smarak
Created on 03-19-2021 12:10 AM - edited 03-19-2021 12:12 AM
Hello @WayneWang
As we haven't received any further Update, We are closing the Post assuming the issue was handled by the Steps shared above [1]. In HBase v1.x, We have limited choices with HBase v2.x using a new AssignmentManager (Details in HBASE-12439), which would assist in managing the RIT without ZooKeeper involvement.
Thanks for using Cloudera Community.
- Smarak