Hbase Cluster automatically stopping the service

[#1]

we are doing a 5 node HBASE cluster with 2 Master node and 5 Regionserver. The Hbase server tent to automatically shutdown frequently while looking into the log its showing zookeeper timeout error , when I look into zookeeper there is no much information . Please suggest

Zookeeper Log

2017-01-18 10:59:26,613 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861] - Client attempting to renew session 0x359acd10a5c0025 at /10.0.0.7:36256
2017-01-18 10:59:26,613 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@108] - Revalidating client: 0x359acd10a5c0025
2017-01-18 10:59:26,616 - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@617] - Established session 0x359acd10a5c0025 with negotiated timeout 40000 for client /10.0.0.7:36256
2017-01-18 13:19:14,671 - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.0.0.7:36256 which had sessionid 0x359acd10a5c0025

Line 1405: 2017-01-18 13:12:30,400 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861] - Client attempting to renew session 0x259ae94b8d2003a at /10.0.0.5:48834
	Line 1406: 2017-01-18 13:12:30,400 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@108] - Revalidating client: 0x259ae94b8d2003a
	Line 1407: 2017-01-18 13:12:30,401 - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@617] - Established session 0x259ae94b8d2003a with negotiated timeout 40000 for client /10.0.0.5:48834
	Line 1928: 2017-01-18 13:32:36,430 - INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.0.0.5:48834 which had sessionid 0x259ae94b8d2003a


Line 1777: 2017-01-18 13:12:30,471 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861] - Client attempting to renew session 0x359acd10a5c0025 at /10.0.0.7:33921
	Line 1778: 2017-01-18 13:12:30,471 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@617] - Established session 0x359acd10a5c0025 with negotiated timeout 40000 for client /10.0.0.7:33921
	Line 1930: 2017-01-18 13:18:10,823 - INFO  [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x359acd10a5c0025 type:setData cxid:0x100 zxid:0x1200006fe4 txntype:-1 reqpath:n/a Error Path:/hbase-unsecure/meta-region-server Error:KeeperErrorCode = NoNode for /hbase-unsecure/meta-region-server
	Line 2411: EndOfStreamException: Unable to read additional data from client sessionid 0x359acd10a5c0025, likely client has closed socket
	Line 2415: 2017-01-18 13:33:09,857 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.0.0.7:33921 which had sessionid 0x359acd10a5c0025
	Line 2833: 2017-01-18 13:33:19,219 - INFO  [SessionTracker:ZooKeeperServer@347] - Expiring session 0x359acd10a5c0025, timeout of 40000ms exceeded
	Line 2848: 2017-01-18 13:33:19,226 - INFO  [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@494] - Processed session termination for sessionid: 0x359acd10a5c0025
	Line 3280: 2017-01-18 13:33:23,031 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861] - Client attempting to renew session 0x359acd10a5c0025 at /10.0.0.7:36164
	Line 3281: 2017-01-18 13:33:23,031 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@610] - Invalid session 0x359acd10a5c0025 for client /10.0.0.7:36164, probably expired
	Line 3282: 2017-01-18 13:33:23,031 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.0.0.7:36164 which had sessionid 0x359acd10a5c0025
Line 243: 2017-01-18 10:59:26,436 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861] - Client attempting to renew session 0x259ae94b8d2003a at /10.0.0.5:52330
	Line 244: 2017-01-18 10:59:26,437 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@108] - Revalidating client: 0x259ae94b8d2003a
	Line 245: 2017-01-18 10:59:26,437 - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@617] - Established session 0x259ae94b8d2003a with negotiated timeout 40000 for client /10.0.0.5:52330
	Line 1946: 2017-01-18 13:19:14,672 - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.0.0.5:52330 which had sessionid 0x259ae94b8d2003a
	Line 2817: 2017-01-18 13:32:50,740 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861] - Client attempting to renew session 0x259ae94b8d2003a at /10.0.0.5:45016
	Line 2818: 2017-01-18 13:32:50,740 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@108] - Revalidating client: 0x259ae94b8d2003a
	Line 3178: 2017-01-18 13:33:08,373 - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.0.0.5:45016 which had sessionid 0x259ae94b8d2003a
	Line 3388: 2017-01-18 13:33:19,024 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861] - Client attempting to renew session 0x259ae94b8d2003a at /10.0.0.5:45160
	Line 3389: 2017-01-18 13:33:19,024 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@108] - Revalidating client: 0x259ae94b8d2003a
	Line 3393: 2017-01-18 13:33:19,224 - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@610] - Invalid session 0x259ae94b8d2003a for client /10.0.0.5:45160, probably expired
	Line 3395: 2017-01-18 13:33:19,224 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.0.0.5:45160 which had sessionid 0x259ae94b8d2003a
Line 1819: 2017-01-18 13:12:50,852 - INFO  [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x259ae94b8d2003a type:create cxid:0x45 zxid:0x12000069c9 txntype:-1 reqpath:n/a Error Path:/hbase-unsecure/flush-table-proc/acquired Error:KeeperErrorCode = NodeExists for /hbase-unsecure/flush-table-proc/acquired
	Line 1820: 2017-01-18 13:12:50,860 - INFO  [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x259ae94b8d2003a type:create cxid:0x4b zxid:0x12000069ca txntype:-1 reqpath:n/a Error Path:/hbase-unsecure/online-snapshot/acquired Error:KeeperErrorCode = NodeExists for /hbase-unsecure/online-snapshot/acquired
	Line 1942: 2017-01-18 13:19:11,171 - INFO  [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x259ae94b8d2003a type:create cxid:0x10c6 zxid:0x1200007115 txntype:-1 reqpath:n/a Error Path:/hbase-unsecure/namespace/default Error:KeeperErrorCode = NodeExists for /hbase-unsecure/namespace/default
	Line 1943: 2017-01-18 13:19:11,184 - INFO  [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x259ae94b8d2003a type:create cxid:0x10c8 zxid:0x1200007117 txntype:-1 reqpath:n/a Error Path:/hbase-unsecure/namespace/hbase Error:KeeperErrorCode = NodeExists for /hbase-unsecure/namespace/hbase
	Line 2302: 2017-01-18 13:33:09,825 - INFO  [SessionTracker:ZooKeeperServer@347] - Expiring session 0x259ae94b8d2003a, timeout of 40000ms exceeded
	Line 2606: 2017-01-18 13:33:14,627 - INFO  [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@494] - Processed session termination for sessionid: 0x259ae94b8d2003a
	Line 3021: 2017-01-18 13:33:19,240 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861] - Client attempting to renew session 0x259ae94b8d2003a at /10.0.0.5:49576
	Line 3022: 2017-01-18 13:33:19,240 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@610] - Invalid session 0x259ae94b8d2003a for client /10.0.0.5:49576, probably expired
	Line 3024: 2017-01-18 13:33:19,240 - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.0.0.5:49576 which had sessionid 0x259ae94b8d2003a

HBase Master Log

2017-01-18 13:33:19,699 INFO  [master/ApacheHadoopMaster1/10.0.0.5:16000] regionserver.HRegionServer: stopping server apachehadoopmaster1,16000,1484704568258; all regions closed.
2017-01-18 13:33:19,699 INFO  [master/ApacheHadoopMaster1/10.0.0.5:16000] hbase.ChoreService: Chore service for: apachehadoopmaster1,16000,1484704568258 had [[ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: apachehadoopmaster1,16000,1484704568258-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: apachehadoopmaster1,16000,1484704568258-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CatalogJanitor-ApacheHadoopMaster1:16000 Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: apachehadoopmaster1,16000,1484704568258-RegionNormalizerChore Period: 1800000 Unit: MILLISECONDS]] on shutdown
2017-01-18 13:33:19,701 WARN  [master/ApacheHadoopMaster1/10.0.0.5:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/master
2017-01-18 13:33:20,702 WARN  [master/ApacheHadoopMaster1/10.0.0.5:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/master
2017-01-18 13:33:20,896 INFO  [ApacheHadoopMaster1:16000.activeMasterManager-SendThread(ApacheHadoopSlave01:2181)] zookeeper.ClientCnxn: Client session timed out, have not heard from server in 15227ms for sessionid 0x359b13b373a0058, closing socket connection and attempting reconnect
2017-01-18 13:33:21,192 INFO  [ApacheHadoopMaster1:16000.activeMasterManager-SendThread(ApacheHadoopSlave01:2181)] zookeeper.ClientCnxn: Session establishment complete on server ApacheHadoopSlave01/10.0.0.7:2181, sessionid = 0x359b13b373a0059, negotiated timeout = 40000
2017-01-18 13:33:21,466 INFO  [ApacheHadoopMaster1:16000.activeMasterManager-SendThread(ApacheHadoopMaster02:2181)] zookeeper.ClientCnxn: Opening socket connection to server ApacheHadoopMaster02/10.0.0.6:2181. Will not attempt to authenticate using SASL (unknown error)
2017-01-18 13:33:21,467 INFO  [ApacheHadoopMaster1:16000.activeMasterManager-SendThread(ApacheHadoopMaster02:2181)] zookeeper.ClientCnxn: Socket connection established to ApacheHadoopMaster02/10.0.0.6:2181, initiating session
2017-01-18 13:33:21,469 INFO  [ApacheHadoopMaster1:16000.activeMasterManager-SendThread(ApacheHadoopMaster02:2181)] zookeeper.ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x359b13b373a0058 has expired, closing socket connection
2017-01-18 13:33:21,469 WARN  [ApacheHadoopMaster1:16000.activeMasterManager-EventThread] client.ConnectionManager$HConnectionImplementation: This client just lost it's session with ZooKeeper, closing it. It will be recreated next time someone needs it
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
	at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:613)
	at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:524)
	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:534)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
2017-01-18 13:33:21,469 INFO  [ApacheHadoopMaster1:16000.activeMasterManager-EventThread] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x359b13b373a0058
2017-01-18 13:33:21,469 INFO  [ApacheHadoopMaster1:16000.activeMasterManager-EventThread] zookeeper.ClientCnxn: EventThread shut down
2017-01-18 13:33:22,648 ERROR [PriorityRpcServer.handler=18,queue=0,port=16000] master.MasterRpcServices: Region server apachehadoopslave03,16020,1484732082248 reported a fatal error:
ABORTING region server apachehadoopslave03,16020,1484732082248: regionserver:16020-0x159b099b501003c, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, baseZNode=/hbase-unsecure regionserver:16020-0x159b099b501003c received expired from ZooKeeper, aborting
Cause:
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
	at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:613)
	at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:524)
	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:534)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)


2017-01-18 13:33:22,702 WARN  [master/ApacheHadoopMaster1/10.0.0.5:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/master
2017-01-18 13:33:23,045 ERROR [PriorityRpcServer.handler=4,queue=0,port=16000] master.MasterRpcServices: Region server apachehadoopslave01,16020,1484663436113 reported a fatal error:
ABORTING region server apachehadoopslave01,16020,1484663436113: regionserver:16020-0x359acd10a5c0025, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, baseZNode=/hbase-unsecure regionserver:16020-0x359acd10a5c0025 received expired from ZooKeeper, aborting
Cause:
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
	at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:613)
	at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:524)
	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:534)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)


2017-01-18 13:33:26,702 WARN  [master/ApacheHadoopMaster1/10.0.0.5:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/master
2017-01-18 13:33:34,702 WARN  [master/ApacheHadoopMaster1/10.0.0.5:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/master
2017-01-18 13:33:34,702 ERROR [master/ApacheHadoopMaster1/10.0.0.5:16000] zookeeper.RecoverableZooKeeper: ZooKeeper getData failed after 4 attempts
2017-01-18 13:33:34,703 WARN  [master/ApacheHadoopMaster1/10.0.0.5:16000] zookeeper.ZKUtil: master:16000-0x259ae94b8d2003a, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, baseZNode=/hbase-unsecure Unable to get data of znode /hbase-unsecure/master
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/master
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:359)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:622)
	at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:148)
	at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:267)
	at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1175)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1073)
	at java.lang.Thread.run(Thread.java:745)
2017-01-18 13:33:34,703 ERROR [master/ApacheHadoopMaster1/10.0.0.5:16000] zookeeper.ZooKeeperWatcher: master:16000-0x259ae94b8d2003a, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, baseZNode=/hbase-unsecure Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/master
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:359)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:622)
	at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:148)
	at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:267)
	at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1175)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1073)
	at java.lang.Thread.run(Thread.java:745)
2017-01-18 13:33:34,703 ERROR [master/ApacheHadoopMaster1/10.0.0.5:16000] master.ActiveMasterManager: master:16000-0x259ae94b8d2003a, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, baseZNode=/hbase-unsecure Error deleting our own master address node
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/master
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:359)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:622)
	at org.apache.hadoop.hbase.zookeeper.MasterAddressTracker.getMasterAddress(MasterAddressTracker.java:148)
	at org.apache.hadoop.hbase.master.ActiveMasterManager.stop(ActiveMasterManager.java:267)
	at org.apache.hadoop.hbase.master.HMaster.stopServiceThreads(HMaster.java:1175)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1073)
	at java.lang.Thread.run(Thread.java:745)
2017-01-18 13:33:34,703 INFO  [master/ApacheHadoopMaster1/10.0.0.5:16000] hbase.ChoreService: Chore service for: apachehadoopmaster1,16000,1484704568258_splitLogManager_ had [] on shutdown
2017-01-18 13:33:34,703 INFO  [master/ApacheHadoopMaster1/10.0.0.5:16000] flush.MasterFlushTableProcedureManager: stop: server shutting down.
2017-01-18 13:33:34,704 INFO  [master/ApacheHadoopMaster1/10.0.0.5:16000] ipc.RpcServer: Stopping server on 16000
2017-01-18 13:33:34,704 INFO  [RpcServer.listener,port=16000] ipc.RpcServer: RpcServer.listener,port=16000: stopping
2017-01-18 13:33:34,705 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped
2017-01-18 13:33:34,705 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping
2017-01-18 13:33:34,708 WARN  [master/ApacheHadoopMaster1/10.0.0.5:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/rs/apachehadoopmaster1,16000,1484704568258
2017-01-18 13:33:35,708 WARN  [master/ApacheHadoopMaster1/10.0.0.5:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/rs/apachehadoopmaster1,16000,1484704568258
2017-01-18 13:33:37,708 WARN  [master/ApacheHadoopMaster1/10.0.0.5:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/rs/apachehadoopmaster1,16000,1484704568258
2017-01-18 13:33:41,708 WARN  [master/ApacheHadoopMaster1/10.0.0.5:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/rs/apachehadoopmaster1,16000,1484704568258
2017-01-18 13:33:49,709 WARN  [master/ApacheHadoopMaster1/10.0.0.5:16000] zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=apachehadoopmaster02:2181,apachehadoopmaster1:2181,apachehadoopslave01:2181, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/rs/apachehadoopmaster1,16000,1484704568258
2017-01-18 13:33:49,709 ERROR [master/ApacheHadoopMaster1/10.0.0.5:16000] zookeeper.RecoverableZooKeeper: ZooKeeper delete failed after 4 attempts
2017-01-18 13:33:49,709 WARN  [master/ApacheHadoopMaster1/10.0.0.5:16000] regionserver.HRegionServer: Failed deleting my ephemeral node
org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase-unsecure/rs/apachehadoopmaster1,16000,1484704568258
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873)
	at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:178)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1222)
	at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1211)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1405)
	at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1081)
	at java.lang.Thread.run(Thread.java:745)
2017-01-18 13:33:49,709 INFO  [master/ApacheHadoopMaster1/10.0.0.5:16000] regionserver.HRegionServer: stopping server apachehadoopmaster1,16000,1484704568258; zookeeper connection closed.
2017-01-18 13:33:49,709 INFO  [master/ApacheHadoopMaster1/10.0.0.5:16000] regionserver.HRegionServer: master/ApacheHadoopMaster1/10.0.0.5:16000 exiting

[#2]

@Anas A

Please have a look at this document

[#3]

Might want to double-check your link 🙂

[#4]

ZooKeeper session expiration happens when the client (HBase RegionServer) fails to successfully contact the ZooKeeper server. This can happen for a variety of reasons:

• JVM GC Pauses in the RegionServer
• Swapping of pages in memory to disk (instead of remaining in memory)
• Network partitions -- the RS host is physically unable to send traffic to the ZK host
• ZooKeeper connection rate-limiting https://community.hortonworks.com/articles/51191/understanding-apache-zookeeper-connection-rate-lim....

Typically, JVM GC pauses and swapping are the most common causes. Make sure that you have adequate memory on your system and configured for the RegionServer. The article linked for ZK connection rate-limiting has instructions to check if that is happening on your system.

If you are a Hortonworks customer, please consider using SmartSense to help automatically diagnose some of these issues.