Support Questions
Find answers, ask questions, and share your expertise

After Upgrading to HDP 3.0.1 Hbase balancer stucks and kill regionserver with Exception org.apache.hadoop.hbase.exceptions.UnexpectedStateException

New Contributor

2018-10-30 18:34:39,920 INFO [RS_CLOSE_REGION-regionserver/foresightdata04stg:16020-1] regionserver.HRegionServer: Adding b8368014c26c480d97b638f6dc47a55b move to foresightdata05stg.region.com,16020,1540898332575 record at close sequenceid=69 2018-10-30 18:34:39,920 INFO [RS_CLOSE_REGION-regionserver/foresightdata04stg:16020-0] regionserver.HRegionServer: Adding 8bf1e947a9d0fa59dbe5edad99b5827b move to foresightdata05stg.region.com,16020,1540898332575 record at close sequenceid=64621176 2018-10-30 18:34:40,871 ERROR [regionserver/foresightdata04stg:16020] regionserver.HRegionServer: ***** ABORTING region server foresightdata04stg.region.com,16020,1540898338206: org.apache.hadoop.hbase.YouAreDeadException: rit=OPEN, location=foresightdata05stg.region.com,16020,1540898332575, table=NVCompititiveBaseLatest, region=b8368014c26c480d97b638f6dc47a55b reported OPEN on server=foresightdata04stg.region.com,16020,1540898338206 but state has otherwise. at org.apache.hadoop.hbase.master.assignment.AssignmentManager.checkOnlineRegionsReport(AssignmentManager.java:1065) at org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportOnlineRegions(AssignmentManager.java:987) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:460) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:13118) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) Caused by: org.apache.hadoop.hbase.exceptions.UnexpectedStateException: rit=OPEN, location=foresightdata05stg.region.com,16020,1540898332575, table=table_test, region=b8368014c26c480d97b638f6dc47a55b reported OPEN on server=foresightdata04stg.region.com,16020,1540898338206 but state has otherwise. at org.apache.hadoop.hbase.master.assignment.AssignmentManager.checkOnlineRegionsReport(AssignmentManager.java:1037) ... 7 more ***** org.apache.hadoop.hbase.YouAreDeadException: org.apache.hadoop.hbase.YouAreDeadException: rit=OPEN, location=foresightdata05stg.region.com,16020,1540898332575, table=NVCompititiveBaseLatest, region=b8368014c26c480d97b638f6dc47a55b reported OPEN on server=foresightdata04stg.region.com,16020,1540898338206 but state has otherwise. at org.apache.hadoop.hbase.master.assignment.AssignmentManager.checkOnlineRegionsReport(AssignmentManager.java:1065) at org.apache.hadoop.hbase.master.assignment.AssignmentManager.reportOnlineRegions(AssignmentManager.java:987) at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:460) at org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:13118) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) @

2 REPLIES 2

New Contributor

Already hbase hbck -metaonly, it's show meta is OK.

New Contributor

Facing same problem...