Options
- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Region stuck in transition, various versions expected
Labels:
- Labels:
-
Apache HBase
Explorer
Created on 05-14-2019 07:34 AM - edited 09-16-2022 07:23 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2019-05-14 09:18:26,042 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of region=deveng_v500,\x00\x00\x1C\xAB\x92\xBC\xD8\x02,1544486155414.ab693aebe203bc8781f1a9f1c0a1d045., starting to roll back the global memstore size. 2019-05-14 09:18:26,043 INFO org.apache.hadoop.hbase.coordination.ZkOpenRegionCoordination: Opening of region {ENCODED => ab693aebe203bc8781f1a9f1c0a1d045, NAME => 'deveng_v500,\x00\x00\x1C\xAB\x92\xBC\xD8\x02,1544486155414.ab693aebe203bc8781f1a9f1c0a1d045.', STARTKEY => '\x00\x00\x1C\xAB\x92\xBC\xD8\x02', ENDKEY => '\x00\x00L\xC6\xAD\xD1\x04'} failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version 40
2019-05-14 09:18:31,562 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of region=deveng_v500,\x00\x00\x1C\xAB\x92\xBC\xD8\x02,1544486155414.ab693aebe203bc8781f1a9f1c0a1d045., starting to roll back the global memstore size. 2019-05-14 09:18:31,562 INFO org.apache.hadoop.hbase.coordination.ZkOpenRegionCoordination: Opening of region {ENCODED => ab693aebe203bc8781f1a9f1c0a1d045, NAME => 'deveng_v500,\x00\x00\x1C\xAB\x92\xBC\xD8\x02,1544486155414.ab693aebe203bc8781f1a9f1c0a1d045.', STARTKEY => '\x00\x00\x1C\xAB\x92\xBC\xD8\x02', ENDKEY => '\x00\x00L\xC6\xAD\xD1\x04'} failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version 58
The region is stuck trying to open on different region servers. I've cycled the nodes to force it to attempt to online elsewhere since the move command doesn't do anything, but no luck. fsck is clean, but hbck with -fixAssignments can't online the region.
19/05/14 09:20:22 WARN util.HBaseFsck: Skip region 'deveng_v500,\x00\x00\x1C\xAB\x92\xBC\xD8\x02,1544486155414.ab693aebe203bc8781f1a9f1c0a1d045.' 19/05/14 09:20:22 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService 19/05/14 09:20:22 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x16a31adec7becec 19/05/14 09:20:22 INFO zookeeper.ZooKeeper: Session: 0x16a31adec7becec closed 19/05/14 09:20:22 INFO zookeeper.ClientCnxn: EventThread shut down Exception in thread "main" java.io.IOException: 1 region(s) could not be checked or repaired. See logs for detail.
2 REPLIES 2
Mentor
Created 05-14-2019 07:37 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Look for an exception in logs preceding the "Failed open of region="
handling failure message on your RegionServer. One situation may be that a
HFile is un-openable under the region (for varied reasons), and will
require being sidelined (removed away) for bringing the region back online.
handling failure message on your RegionServer. One situation may be that a
HFile is un-openable under the region (for varied reasons), and will
require being sidelined (removed away) for bringing the region back online.
Explorer
Created 05-15-2019 07:17 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Before the failure to open message is this block (only including first line of java stackstrace):
2019-05-14 15:55:53,356 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://athos/hbase/data/default/deveng_v500/ab693aebe203bc8781f1a9f1c0a1d045/recovered.edits/0000000000094270192 2019-05-14 15:55:53,383 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://athos/hbase/data/default/deveng_v500/ab693aebe203bc8781f1a9f1c0a1d045/recovered.edits/0000000000094270299 2019-05-14 15:55:53,722 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://athos/hbase/data/default/deveng_v500/ab693aebe203bc8781f1a9f1c0a1d045/recovered.edits/0000000000094270330 2019-05-14 15:55:53,903 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Auth successful for tomcat (auth:SIMPLE) 2019-05-14 15:55:53,904 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Connection from 10.190.158.151 port: 60648 with unknown version info 2019-05-14 15:55:54,614 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of region=deveng_v500,\x00\x00\x1C\xAB\x92\xBC\xD8\x02,1544486155414.ab693aebe203bc8781f1a9f1c0a1d045., starting to roll back the global memstore size. java.lang.IllegalArgumentException: offset (8) + length (2) exceed the capacity of the array: 0 at org.apache.hadoop.hbase.util.Bytes.explainWrongLengthOrOffset(Bytes.java:631) .............
