Member since
11-24-2020
3
Posts
0
Kudos Received
0
Solutions
02-07-2021
06:24 PM
Hi,
we have a 3 datanode hadoop cluster (CDH 5.10). the replication factor is set to 2. now we cannot access data from Hbase table due to failed regions on that table. could somebody can give me some suggestion to fix this issue(we don't lost data)?
the failed regions are in "Regions in Transition" state. we tried to fix and repair the table by below command.
$ hbase hbck -fix <table>
$ hbase hbck -repair <table>
But failed, we got error like below:
21/02/08 02:03:33 INFO util.HBaseFsck: Checking and fixing region consistency
ERROR: Region { meta => xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952., hdfs => hdfs://cdh-cm.test.com:8020/hbase/data/default/xxxxLogsIndexer/327a6ed6b00c95e63346f6d725147952, deployed => , replicaId => 0 } not deployed on any region server.
Trying to fix unassigned region...
21/02/08 02:03:33 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'}
21/02/08 02:03:34 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'}
21/02/08 02:03:35 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'}
21/02/08 02:03:36 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'}
21/02/08 02:03:37 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'}
21/02/08 02:03:38 INFO util.HBaseFsckRepair: Region still in transition, waiting for it to become assigned: {ENCODED => 327a6ed6b00c95e63346f6d725147952, NAME => 'xxxxLogsIndexer,,1588737167917.327a6ed6b00c95e63346f6d725147952.', STARTKEY => '', ENDKEY => '\x00:\x7F\xFF\xFE\x8Eb\xAAY\xF4:Z\x891BlAF\x93\x98Q^\x1F\xAF\xA8\xEE\x97'}
^C21/02/08 02:03:39 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
21/02/08 02:03:39 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x1777c1d848d09be
21/02/08 02:03:39 INFO zookeeper.ZooKeeper: Session: 0x1777c1d848d09be closed
21/02/08 02:03:39 INFO zookeeper.ClientCnxn: EventThread shut down
21/02/08 02:03:39 INFO util.HBaseFsck: Finishing hbck
Thanks!
Br, Wayne
... View more
Labels:
01-11-2021
01:34 AM
same issue here with default configuration (once in 7 days). any suggestions? 2021-01-11 03:27:19,028 INFO org.apache.hadoop.hbase.regionserver.HStore: Completed major compaction of 4 (all) file(s) in Info of CallLogs,\xE9\x7F\x9EJ\x10\x06L\xF7\x9A\xBF+\xCD\xA8\xB7\x9D\xBB,1608101963434.f0d1a5f4e816118ac167fe9730258102. into a54298ecc9594f9aa0cf6657a795bb54(size=6.0 G), total size for store is 6.0 G. This selection was in queue for 0sec, and took 56sec to execute.
2021-01-11 03:55:46,943 INFO org.apache.hadoop.hbase.regionserver.HStore: Completed major compaction of 5 (all) file(s) in Info of CallLogs,M}\xC4;,1609043919090.8c704f7385c3c3b07bc3aa4be1adc577. into 1d3f086c88bc4972a2e550dd093e9824(size=5.7 G), total size for store is 5.7 G. This selection was in queue for 0sec, and took 1mins, 29sec to execute.
2021-01-11 04:24:56,056 INFO org.apache.hadoop.hbase.regionserver.HStore: Completed major compaction of 4 (all) file(s) in Info of CallLogs,.\x81\xC6\x99e1K\x00\xAE\xB3@\x14g \x0Av,1608158031158.4998d8db979dfea2751136bf1767fb1b. into 4b1c6db0ed5d440d9adb58bf00109b57(size=5.6 G), total size for store is 5.6 G. This selection was in queue for 0sec, and took 1mins, 33sec to execute.
2021-01-11 05:36:34,562 INFO org.apache.hadoop.hbase.regionserver.HStore: Completed major compaction of 5 (all) file(s) in Info of CallLogs,\x19~A\x8F\xD3^G\xFB\xB5!.\x8C6\xCB\xC7t,1607673667302.6c08c1a2f5648c5f190bc378f628a838. into 71d8c8268fcc46d4ad1be29a6c6ce880(size=5.9 G), total size for store is 5.9 G. This selection was in queue for 0sec, and took 1mins, 42sec to execute.
2021-01-11 05:38:13,268 INFO org.apache.hadoop.hbase.regionserver.HStore: Completed major compaction of 4 (all) file(s) in Info of CallLogs,\xB9~\x8EX\xA5cH\xBE\x94g\xFF\xB76\xD6\x80/,1608131740376.47868b4d2475ef1fef1f23fea51b2e0f. into 21e524b8483047b7a6529ff20ea56602(size=5.9 G), total size for store is 5.9 G. This selection was in queue for 0sec, and took 1mins, 17sec to execute.
2021-01-11 07:11:21,277 INFO org.apache.hadoop.hbase.regionserver.HStore: Completed major compaction of 3 (all) file(s) in Info of CallLogs,\x9E\x7F\xD9\xADe\x81H\x8C\x8E\x80\x87)\xE0G\xD7\xFE,1608533709336.c152e8f046c8f75ca265c1fc9c742909. into f4dc8f12dcee4f118acc779ae000ff6b(size=5.8 G), total size for store is 5.8 G. This selection was in queue for 0sec, and took 1mins, 19sec to execute.
2021-01-11 08:51:46,548 INFO org.apache.hadoop.hbase.regionserver.HStore: Completed major compaction of 5 (all) file(s) in Info of CallLogs,\x7F\x81\x01M8{O\x92\x9C\xC2J\x01\xB7r8\xF4,1608529936723.34e0ffe299134cd8ad22145ae2314d3e. into 246537faa92741ea865437eebf2e1e9a(size=6.0 G), total size for store is 6.0 G. This selection was in queue for 0sec, and took 1mins, 29sec to execute.
... View more
11-24-2020
12:22 AM
Hi,
I have one question about the impact of change exist rack id.
we have a CDH 5 cluster with 6 data nodes( replication factor is 3) , they are belong one rack rack0.
and we add 1 data node to this cluster, but it is added rack rack1 by mistake.
now we met issue "Under-Replicated Blocks" due to only 1 node in rack1, it is need replicate 1 of 3 to node in rack1. but the disk capacity is not enough to storage 1 replica of all files.
Now, we have decomissioned the node in rack1, and want to change the node back to rack0.
my question is what is the steps to change the node back to rack0 safely? any suggestions?
Thanks!
... View more
Labels:
