Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Region stuck in transition, various versions expected

avatar
Explorer
2019-05-14 09:18:26,042 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of region=deveng_v500,\x00\x00\x1C\xAB\x92\xBC\xD8\x02,1544486155414.ab693aebe203bc8781f1a9f1c0a1d045., starting to roll back the global memstore size.
2019-05-14 09:18:26,043 INFO org.apache.hadoop.hbase.coordination.ZkOpenRegionCoordination: Opening of region {ENCODED => ab693aebe203bc8781f1a9f1c0a1d045, NAME => 'deveng_v500,\x00\x00\x1C\xAB\x92\xBC\xD8\x02,1544486155414.ab693aebe203bc8781f1a9f1c0a1d045.', STARTKEY => '\x00\x00\x1C\xAB\x92\xBC\xD8\x02', ENDKEY => '\x00\x00L\xC6\xAD\xD1\x04'} failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version 40
2019-05-14 09:18:31,562 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of region=deveng_v500,\x00\x00\x1C\xAB\x92\xBC\xD8\x02,1544486155414.ab693aebe203bc8781f1a9f1c0a1d045., starting to roll back the global memstore size.
2019-05-14 09:18:31,562 INFO org.apache.hadoop.hbase.coordination.ZkOpenRegionCoordination: Opening of region {ENCODED => ab693aebe203bc8781f1a9f1c0a1d045, NAME => 'deveng_v500,\x00\x00\x1C\xAB\x92\xBC\xD8\x02,1544486155414.ab693aebe203bc8781f1a9f1c0a1d045.', STARTKEY => '\x00\x00\x1C\xAB\x92\xBC\xD8\x02', ENDKEY => '\x00\x00L\xC6\xAD\xD1\x04'} failed, transitioning from OPENING to FAILED_OPEN in ZK, expecting version 58

The region is stuck trying to open on different region servers. I've cycled the nodes to force it to attempt to online elsewhere since the move command doesn't do anything, but no luck. fsck is clean, but hbck with -fixAssignments can't online the region. 

 

19/05/14 09:20:22 WARN util.HBaseFsck: Skip region 'deveng_v500,\x00\x00\x1C\xAB\x92\xBC\xD8\x02,1544486155414.ab693aebe203bc8781f1a9f1c0a1d045.'
19/05/14 09:20:22 INFO client.ConnectionManager$HConnectionImplementation: Closing master protocol: MasterService
19/05/14 09:20:22 INFO client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x16a31adec7becec
19/05/14 09:20:22 INFO zookeeper.ZooKeeper: Session: 0x16a31adec7becec closed
19/05/14 09:20:22 INFO zookeeper.ClientCnxn: EventThread shut down
Exception in thread "main" java.io.IOException: 1 region(s) could not be checked or repaired.  See logs for detail.
2 REPLIES 2

avatar
Mentor
Look for an exception in logs preceding the "Failed open of region="
handling failure message on your RegionServer. One situation may be that a
HFile is un-openable under the region (for varied reasons), and will
require being sidelined (removed away) for bringing the region back online.

avatar
Explorer

Before the failure to open message is this block (only including first line of java stackstrace):

 

2019-05-14 15:55:53,356 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://athos/hbase/data/default/deveng_v500/ab693aebe203bc8781f1a9f1c0a1d045/recovered.edits/0000000000094270192
2019-05-14 15:55:53,383 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://athos/hbase/data/default/deveng_v500/ab693aebe203bc8781f1a9f1c0a1d045/recovered.edits/0000000000094270299
2019-05-14 15:55:53,722 INFO org.apache.hadoop.hbase.regionserver.HRegion: Replaying edits from hdfs://athos/hbase/data/default/deveng_v500/ab693aebe203bc8781f1a9f1c0a1d045/recovered.edits/0000000000094270330
2019-05-14 15:55:53,903 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Auth successful for tomcat (auth:SIMPLE)
2019-05-14 15:55:53,904 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Connection from 10.190.158.151 port: 60648 with unknown version info
2019-05-14 15:55:54,614 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of region=deveng_v500,\x00\x00\x1C\xAB\x92\xBC\xD8\x02,1544486155414.ab693aebe203bc8781f1a9f1c0a1d045., starting to roll back the global memstore size.
java.lang.IllegalArgumentException: offset (8) + length (2) exceed the capacity of the array: 0
	at org.apache.hadoop.hbase.util.Bytes.explainWrongLengthOrOffset(Bytes.java:631)
.............