Created on 05-14-2019 07:34 AM - edited 09-16-2022 07:23 AM
2019-05-14 09:18:26,042 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of region=deveng_v500,\x00\x00\x1C\xAB\x92\xBC\xD8\x02,1544486155414.ab693aebe203bc8781f1a9f1c0a1d045., starting to roll back the global memstore size. 2019-05-14 09:18:26,043 INFO org.apache.hadoop.hbase.coordination.ZkOpenRegionCoordination: Opening of region {ENCODED => ab693aebe203bc8781f1a9f1c0a1d045, NAME => 'deveng_v500,\x00\x00\x1C\xAB\x92\xBC\xD8\x02,1544486155414.ab693aebe203bc8781f1a9f1c0a1d045.', STARTKEY => '\x00\x00\x1C\xAB\x92\xBC\xD8\x02', ENDKEY => '\x00\x00L\xC6\xAD\xD1\x04'} failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version 40
2019-05-14 09:18:31,562 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of region=deveng_v500,\x00\x00\x1C\xAB\x92\xBC\xD8\x02,1544486155414.ab693aebe203bc8781f1a9f1c0a1d045., starting to roll back the global memstore size. 2019-05-14 09:18:31,562 INFO org.apache.hadoop.hbase.coordination.ZkOpenRegionCoordination: Opening of region {ENCODED => ab693aebe203bc8781f1a9f1c0a1d045, NAME => 'deveng_v500,\x00\x00\x1C\xAB\x92\xBC\xD8\x02,1544486155414.ab693aebe203bc8781f1a9f1c0a1d045.', STARTKEY => '\x00\x00\x1C\xAB\x92\xBC\xD8\x02', ENDKEY => '\x00\x00L\xC6\xAD\xD1\x04'} failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version 58
The region is stuck trying to open on different region servers. I've cycled the nodes to force it to attempt to online elsewhere since the move command doesn't do anything, but no luck. fsck is clean, but hbck with -fixAssignments can't online the region.
19/05/14 09:20:22 WARN util.HBaseFsck: Skip region 'deveng_v500,\x00\x00\x1C\xAB\x92\xBC\xD8\x02,1544486155414.ab693aebe203bc8781f1a9f1c0a1d045.' 19/05/14 09:20:22 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService 19/05/14 09:20:22 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x16a31adec7becec 19/05/14 09:20:22 INFO zookeeper.ZooKeeper: Session: 0x16a31adec7becec closed 19/05/14 09:20:22 INFO zookeeper.ClientCnxn: EventThread shut down Exception in thread "main" java.io.IOException: 1 region(s) could not be checked or repaired. See logs for detail.
Created 05-14-2019 07:37 PM
Created 05-15-2019 07:17 AM
Before the failure to open message is this block (only including first line of java stackstrace):
2019-05-14 15:55:53,356 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://athos/hbase/data/default/deveng_v500/ab693aebe203bc8781f1a9f1c0a1d045/recovered.edits/0000000000094270192 2019-05-14 15:55:53,383 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://athos/hbase/data/default/deveng_v500/ab693aebe203bc8781f1a9f1c0a1d045/recovered.edits/0000000000094270299 2019-05-14 15:55:53,722 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://athos/hbase/data/default/deveng_v500/ab693aebe203bc8781f1a9f1c0a1d045/recovered.edits/0000000000094270330 2019-05-14 15:55:53,903 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Auth successful for tomcat (auth:SIMPLE) 2019-05-14 15:55:53,904 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Connection from 10.190.158.151 port: 60648 with unknown version info 2019-05-14 15:55:54,614 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of region=deveng_v500,\x00\x00\x1C\xAB\x92\xBC\xD8\x02,1544486155414.ab693aebe203bc8781f1a9f1c0a1d045., starting to roll back the global memstore size. java.lang.IllegalArgumentException: offset (8) + length (2) exceed the capacity of the array: 0 at org.apache.hadoop.hbase.util.Bytes.explainWrongLengthOrOffset(Bytes.java:631) .............