Member since
02-21-2017
1
Post
0
Kudos Received
0
Solutions
02-21-2017
09:52 AM
Hello team, I have a problem with region server aborted. Part of logs is attached: 2017-02-21 14:32:41,017 FATAL [main-EventThread] regionserver.HRegionServer: ABORTING region server serv01,16020,1486954210496:
regionserver:16020-0x35a173e7b40003f, quorum=serv02.localdomain:2181,serv03.localdomain:2181,serv01.localdomain:
2181, baseZNode=/hbase-unsecure regionserver:16020-0x35a173e7b40003f received expired from ZooKeeper, aborting
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:610)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:521)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:534)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
2017-02-21 14:32:43,438 FATAL [regionserver/serv01/10.0.7.1:16020] regionserver.HRegionServer: ABORTING region server server01,16020,1486954210496: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing serv01,1
6020,1486954210496 as dead server
at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:413)
at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:318)
at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:325)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionSe
rverStatusProtos.java:8617)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
org.apache.hadoop.hbase.YouAreDeadException: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing
serv01,16020,1486954210496 as dead server
at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:413)
at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:318)
at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:325)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionSe
rverStatusProtos.java:8617)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
2017-02-21 14:32:43,879 INFO [regionserver/serv01/10.0.7.1:16020] regionserver.HRegionServer: STOPPED: org.apache.hadoop.hb
ase.YouAreDeadException: Server REPORT rejected; currently processing serv01,16020,1486954210496 as dead server
at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:413)
at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:318)
at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerReport(MasterRpcServices.java:325)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionSe
rverStatusProtos.java:8617)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
2017-02-21 14:33:05,801 ERROR [regionserver/serv01/10.0.7.1:16020] zookeeper.ZooKeeperWatcher: regionserver:16020-0x35a173e7
b40003f, quorum=serv02.localdomain:2181,serv03.localdomain:2181,serv01.localdomain:2181, baseZNode=/hbase-unsecur
e Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/replication/rs/serv01,16020,1486954210496
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:295)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchForNewChildren(ZKUtil.java:454)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenAndWatchThem(ZKUtil.java:482)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenBFSAndWatchThem(ZKUtil.java:1461)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursivelyMultiOrSequential(ZKUtil.java:1383)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeRecursively(ZKUtil.java:1265)
at org.apache.hadoop.hbase.replication.ReplicationQueuesZKImpl.removeAllQueues(ReplicationQueuesZKImpl.java:187)
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.join(ReplicationSourceManager.java:292)
at org.apache.hadoop.hbase.replication.regionserver.Replication.join(Replication.java:180)
at org.apache.hadoop.hbase.replication.regionserver.Replication.stopReplicationService(Replication.java:172)
at org.apache.hadoop.hbase.regionserver.HRegionServer.stopServiceThreads(HRegionServer.java:2137)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1071)
at java.lang.Thread.run(Thread.java:745) Please help.
... View more
Labels: