we are doing a 5 node HBASE cluster with 2 Master node and 5 Regionserver. The Hbase server tent to automatically shutdown frequently while looking into the log its showing zookeeper timeout error , when I look into zookeeper there is no much information . Please suggest Zookeeper Log 2017-01-18 10:59:26,613 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861] - Client attempting to renew session 0x359acd10a5c0025 at /10.0.0.7:36256 2017-01-18 10:59:26,613 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@108] - Revalidating client: 0x359acd10a5c0025 2017-01-18 10:59:26,616 - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@617] - Established session 0x359acd10a5c0025 with negotiated timeout 40000 for client /10.0.0.7:36256 2017-01-18 13:19:14,671 - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.0.0.7:36256 which had sessionid 0x359acd10a5c0025 Line 1405: 2017-01-18 13:12:30,400 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861] - Client attempting to renew session 0x259ae94b8d2003a at /10.0.0.5:48834 Line 1406: 2017-01-18 13:12:30,400 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@108] - Revalidating client: 0x259ae94b8d2003a Line 1407: 2017-01-18 13:12:30,401 - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@617] - Established session 0x259ae94b8d2003a with negotiated timeout 40000 for client /10.0.0.5:48834 Line 1928: 2017-01-18 13:32:36,430 - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.0.0.5:48834 which had sessionid 0x259ae94b8d2003a Line 1777: 2017-01-18 13:12:30,471 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861] - Client attempting to renew session 0x359acd10a5c0025 at /10.0.0.7:33921 Line 1778: 2017-01-18 13:12:30,471 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@617] - Established session 0x359acd10a5c0025 with negotiated timeout 40000 for client /10.0.0.7:33921 Line 1930: 2017-01-18 13:18:10,823 - INFO [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x359acd10a5c0025 type:setData cxid:0x100 zxid:0x1200006fe4 txntype:-1 reqpath:n/a Error Path:/hbase-unsecure/meta-region-server Error:KeeperErrorCode = NoNode for /hbase-unsecure/meta-region-server Line 2411: EndOfStreamException: Unable to read additional data from client sessionid 0x359acd10a5c0025, likely client has closed socket Line 2415: 2017-01-18 13:33:09,857 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.0.0.7:33921 which had sessionid 0x359acd10a5c0025 Line 2833: 2017-01-18 13:33:19,219 - INFO [SessionTracker:ZooKeeperServer@347] - Expiring session 0x359acd10a5c0025, timeout of 40000ms exceeded Line 2848: 2017-01-18 13:33:19,226 - INFO [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@494] - Processed session termination for sessionid: 0x359acd10a5c0025 Line 3280: 2017-01-18 13:33:23,031 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861] - Client attempting to renew session 0x359acd10a5c0025 at /10.0.0.7:36164 Line 3281: 2017-01-18 13:33:23,031 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@610] - Invalid session 0x359acd10a5c0025 for client /10.0.0.7:36164, probably expired Line 3282: 2017-01-18 13:33:23,031 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.0.0.7:36164 which had sessionid 0x359acd10a5c0025 Line 243: 2017-01-18 10:59:26,436 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861] - Client attempting to renew session 0x259ae94b8d2003a at /10.0.0.5:52330 Line 244: 2017-01-18 10:59:26,437 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@108] - Revalidating client: 0x259ae94b8d2003a Line 245: 2017-01-18 10:59:26,437 - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@617] - Established session 0x259ae94b8d2003a with negotiated timeout 40000 for client /10.0.0.5:52330 Line 1946: 2017-01-18 13:19:14,672 - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.0.0.5:52330 which had sessionid 0x259ae94b8d2003a Line 2817: 2017-01-18 13:32:50,740 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861] - Client attempting to renew session 0x259ae94b8d2003a at /10.0.0.5:45016 Line 2818: 2017-01-18 13:32:50,740 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@108] - Revalidating client: 0x259ae94b8d2003a Line 3178: 2017-01-18 13:33:08,373 - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.0.0.5:45016 which had sessionid 0x259ae94b8d2003a Line 3388: 2017-01-18 13:33:19,024 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861] - Client attempting to renew session 0x259ae94b8d2003a at /10.0.0.5:45160 Line 3389: 2017-01-18 13:33:19,024 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@108] - Revalidating client: 0x259ae94b8d2003a Line 3393: 2017-01-18 13:33:19,224 - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@610] - Invalid session 0x259ae94b8d2003a for client /10.0.0.5:45160, probably expired Line 3395: 2017-01-18 13:33:19,224 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.0.0.5:45160 which had sessionid 0x259ae94b8d2003a Line 1819: 2017-01-18 13:12:50,852 - INFO [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x259ae94b8d2003a type:create cxid:0x45 zxid:0x12000069c9 txntype:-1 reqpath:n/a Error Path:/hbase-unsecure/flush-table-proc/acquired Error:KeeperErrorCode = NodeExists for /hbase-unsecure/flush-table-proc/acquired Line 1820: 2017-01-18 13:12:50,860 - INFO [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x259ae94b8d2003a type:create cxid:0x4b zxid:0x12000069ca txntype:-1 reqpath:n/a Error Path:/hbase-unsecure/online-snapshot/acquired Error:KeeperErrorCode = NodeExists for /hbase-unsecure/online-snapshot/acquired Line 1942: 2017-01-18 13:19:11,171 - INFO [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x259ae94b8d2003a type:create cxid:0x10c6 zxid:0x1200007115 txntype:-1 reqpath:n/a Error Path:/hbase-unsecure/namespace/default Error:KeeperErrorCode = NodeExists for /hbase-unsecure/namespace/default Line 1943: 2017-01-18 13:19:11,184 - INFO [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x259ae94b8d2003a type:create cxid:0x10c8 zxid:0x1200007117 txntype:-1 reqpath:n/a Error Path:/hbase-unsecure/namespace/hbase Error:KeeperErrorCode = NodeExists for /hbase-unsecure/namespace/hbase Line 2302: 2017-01-18 13:33:09,825 - INFO [SessionTracker:ZooKeeperServer@347] - Expiring session 0x259ae94b8d2003a, timeout of 40000ms exceeded Line 2606: 2017-01-18 13:33:14,627 - INFO [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@494] - Processed session termination for sessionid: 0x259ae94b8d2003a Line 3021: 2017-01-18 13:33:19,240 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861] - Client attempting to renew session 0x259ae94b8d2003a at /10.0.0.5:49576 Line 3022: 2017-01-18 13:33:19,240 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@610] - Invalid session 0x259ae94b8d2003a for client /10.0.0.5:49576, probably expired Line 3024: 2017-01-18 13:33:19,240 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.0.0.5:49576 which had sessionid 0x259ae94b8d2003a HBase Master Log 2017-01-18 13:33:19,699 INFO [master/ApacheHadoopMaster1/10.0.0.5:16000] regionserver.HRegionServer: stopping server apachehadoopmaster1,16000,1484704568258; all regions closed. 2017-01-18 13:33:19,699 INFO [master/ApacheHadoopMaster1/10.0.0.5:16000] hbase.ChoreService: Chore service for: apachehadoopmaster1,16000,1484704568258 had [[ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: apachehadoopmaster1,16000,1484704568258-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: apachehadoopmaster1,16000,1484704568258-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CatalogJanitor-ApacheHadoopMaster1:16000 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: apachehadoopmaster1,16000,1484704568258-RegionNormalizerChore Period: 1800000 Unit: MILLISECONDS]] on shutdown 2017-01-18 13:33:19,701 WARN [master/ApacheHadoopMaster1/10.0.0.5:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/master 2017-01-18 13:33:20,702 WARN [master/ApacheHadoopMaster1/10.0.0.5:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/master 2017-01-18 13:33:20,896 INFO [ApacheHadoopMaster1:16000.activeMasterManager-SendThread(ApacheHadoopSlave01:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 15227ms for sessionid 0x359b13b373a0058, closing socket connection and attempting reconnect 2017-01-18 13:33:21,192 INFO [ApacheHadoopMaster1:16000.activeMasterManager-SendThread(ApacheHadoopSlave01:2181)] zookeeper.ClientCnxn: Session establishment complete on server ApacheHadoopSlave01/10.0.0.7:2181, sessionid = 0x359b13b373a0059, negotiated timeout = 40000 2017-01-18 13:33:21,466 INFO [ApacheHadoopMaster1:16000.activeMasterManager-SendThread(ApacheHadoopMaster02:2181)] zookeeper.ClientCnxn: Opening socket connection to server ApacheHadoopMaster02/10.0.0.6:2181. Will not attempt to authenticate using SASL (unknown error) 2017-01-18 13:33:21,467 INFO [ApacheHadoopMaster1:16000.activeMasterManager-SendThread(ApacheHadoopMaster02:2181)] zookeeper.ClientCnxn: Socket connection established to ApacheHadoopMaster02/10.0.0.6:2181, initiating session 2017-01-18 13:33:21,469 INFO [ApacheHadoopMaster1:16000.activeMasterManager-SendThread(ApacheHadoopMaster02:2181)] zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x359b13b373a0058 has expired, closing socket connection 2017-01-18 13:33:21,469 WARN [ApacheHadoopMaster1:16000.activeMasterManager-EventThread] client.ConnectionManager$HConnectionImplementation: This client just lost it's session with ZooKeeper, closing it. It will be recreated next time someone needs it org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:613) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:524) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:534) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2017-01-18 13:33:21,469 INFO [ApacheHadoopMaster1:16000.activeMasterManager-EventThread] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x359b13b373a0058 2017-01-18 13:33:21,469 INFO [ApacheHadoopMaster1:16000.activeMasterManager-EventThread] zookeeper.ClientCnxn: EventThread shut down 2017-01-18 13:33:22,648 ERROR [PriorityRpcServer.handler=18,queue=0,port=16000] master.MasterRpcServices: Region server apachehadoopslave03,16020,1484732082248 reported a fatal error: ABORTING region server apachehadoopslave03,16020,1484732082248: regionserver:16020-0x159b099b501003c, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, baseZNode=/hbase-unsecure regionserver:16020-0x159b099b501003c received expired from ZooKeeper, aborting Cause: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:613) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:524) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:534) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2017-01-18 13:33:22,702 WARN [master/ApacheHadoopMaster1/10.0.0.5:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/master 2017-01-18 13:33:23,045 ERROR [PriorityRpcServer.handler=4,queue=0,port=16000] master.MasterRpcServices: Region server apachehadoopslave01,16020,1484663436113 reported a fatal error: ABORTING region server apachehadoopslave01,16020,1484663436113: regionserver:16020-0x359acd10a5c0025, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, baseZNode=/hbase-unsecure regionserver:16020-0x359acd10a5c0025 received expired from ZooKeeper, aborting Cause: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:613) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:524) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:534) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2017-01-18 13:33:26,702 WARN [master/ApacheHadoopMaster1/10.0.0.5:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/master 2017-01-18 13:33:34,702 WARN [master/ApacheHadoopMaster1/10.0.0.5:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/master 2017-01-18 13:33:34,702 ERROR [master/ApacheHadoopMaster1/10.0.0.5:16000] zookeeper.RecoverableZooKeeper: ZooKeeper getData failed after 4 attempts 2017-01-18 13:33:34,703 WARN [master/ApacheHadoopMaster1/10.0.0.5:16000] zookeeper.ZKUtil: master:16000-0x259ae94b8d2003a, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, baseZNode=/hbase-unsecure Unable to get data of znode /hbase-unsecure/master org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:359) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:622) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:148) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:267) at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1175) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1073) at java.lang.Thread.run(Thread.java:745) 2017-01-18 13:33:34,703 ERROR [master/ApacheHadoopMaster1/10.0.0.5:16000] zookeeper.ZooKeeperWatcher: master:16000-0x259ae94b8d2003a, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, baseZNode=/hbase-unsecure Received unexpected KeeperException, re-throwing exception org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:359) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:622) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:148) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:267) at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1175) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1073) at java.lang.Thread.run(Thread.java:745) 2017-01-18 13:33:34,703 ERROR [master/ApacheHadoopMaster1/10.0.0.5:16000] master.ActiveMasterManager: master:16000-0x259ae94b8d2003a, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, baseZNode=/hbase-unsecure Error deleting our own master address node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/master at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:359) at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:622) at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:148) at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:267) at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1175) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1073) at java.lang.Thread.run(Thread.java:745) 2017-01-18 13:33:34,703 INFO [master/ApacheHadoopMaster1/10.0.0.5:16000] hbase.ChoreService: Chore service for: apachehadoopmaster1,16000,1484704568258_splitLogManager_ had [] on shutdown 2017-01-18 13:33:34,703 INFO [master/ApacheHadoopMaster1/10.0.0.5:16000] flush.MasterFlushTableProcedureManager: stop: server shutting down. 2017-01-18 13:33:34,704 INFO [master/ApacheHadoopMaster1/10.0.0.5:16000] ipc.RpcServer: Stopping server on 16000 2017-01-18 13:33:34,704 INFO [RpcServer.listener,port=16000] ipc.RpcServer: RpcServer.listener,port=16000: stopping 2017-01-18 13:33:34,705 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped 2017-01-18 13:33:34,705 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping 2017-01-18 13:33:34,708 WARN [master/ApacheHadoopMaster1/10.0.0.5:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/rs/apachehadoopmaster1,16000,1484704568258 2017-01-18 13:33:35,708 WARN [master/ApacheHadoopMaster1/10.0.0.5:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/rs/apachehadoopmaster1,16000,1484704568258 2017-01-18 13:33:37,708 WARN [master/ApacheHadoopMaster1/10.0.0.5:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/rs/apachehadoopmaster1,16000,1484704568258 2017-01-18 13:33:41,708 WARN [master/ApacheHadoopMaster1/10.0.0.5:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/rs/apachehadoopmaster1,16000,1484704568258 2017-01-18 13:33:49,709 WARN [master/ApacheHadoopMaster1/10.0.0.5:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/rs/apachehadoopmaster1,16000,1484704568258 2017-01-18 13:33:49,709 ERROR [master/ApacheHadoopMaster1/10.0.0.5:16000] zookeeper.RecoverableZooKeeper: ZooKeeper delete failed after 4 attempts 2017-01-18 13:33:49,709 WARN [master/ApacheHadoopMaster1/10.0.0.5:16000] regionserver.HRegionServer: Failed deleting my ephemeral node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/rs/apachehadoopmaster1,16000,1484704568258 at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:178) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1222) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1211) at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1405) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1081) at java.lang.Thread.run(Thread.java:745) 2017-01-18 13:33:49,709 INFO [master/ApacheHadoopMaster1/10.0.0.5:16000] regionserver.HRegionServer: stopping server apachehadoopmaster1,16000,1484704568258; zookeeper connection closed. 2017-01-18 13:33:49,709 INFO [master/ApacheHadoopMaster1/10.0.0.5:16000] regionserver.HRegionServer: master/ApacheHadoopMaster1/10.0.0.5:16000 exiting ... View more