Member since
11-24-2020
3
Posts
0
Kudos Received
0
Solutions
02-07-2021
06:24 PM
Hi,
we have a 3 datanode hadoop cluster (CDH 5.10). the replication factor is set to 2. now we cannot access data from Hbase table due to failed regions on that table. could somebody can give me some suggestion to fix this issue(we don't lost data)?
the failed regions are in "Regions in Transition" state. we tried to fix and repair the table by below command.
$ hbase hbck -fix <table>
$ hbase hbck -repair <table>
But failed, we got error like below:
21/02/08 02:03:33 INFO util.HBaseFsck: Checking and fixing region consistency
ERROR: Region { meta => xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952., hdfs => hdfs://cdh-cm.test.com:8020/hbase/data/default/xxxxLogsIndexer/327a6ed6b00c95e63346f6d725147952, deployed => , replicaId => 0 } not deployed on any region server.
Trying to fix unassigned region...
21/02/08 02:03:33 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'}
21/02/08 02:03:34 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'}
21/02/08 02:03:35 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'}
21/02/08 02:03:36 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'}
21/02/08 02:03:37 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'}
21/02/08 02:03:38 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'}
^C21/02/08 02:03:39 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
21/02/08 02:03:39 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x1777c1d848d09be
21/02/08 02:03:39 INFO zookeeper.ZooKeeper: Session: 0x1777c1d848d09be closed
21/02/08 02:03:39 INFO zookeeper.ClientCnxn: EventThread shut down
21/02/08 02:03:39 INFO util.HBaseFsck: Finishing hbck
Thanks!
Br, Wayne
... View more
Labels:
11-24-2020
12:22 AM
Hi,
I have one question about the impact of change exist rack id.
we have a CDH 5 cluster with 6 data nodes(replication factor is 3), they are belong one rack rack0.
and we add 1 data node to this cluster, but it is added rack rack1 by mistake.
now we met issue "Under-Replicated Blocks" due to only 1 node in rack1, it is need replicate 1 of 3 to node in rack1. but the disk capacity is not enough to storage 1 replica of all files.
Now, we have decomissioned the node in rack1, and want to change the node back to rack0.
my question is what is the steps to change the node back to rack0 safely? any suggestions?
Thanks!
... View more
Labels: