hi all In my HDP cluster, I install 3 zookeeper-servers and zookeeper client on 3 nodes ( master1 , master2 , master3 ) ,. all nodes are on redhat machine version 7.2 When we run the zookeeper-client from master1 on the zookeeper server on master1 , we get CONNECTING When we run the zookeeper-client from master1 on the zookeeper server on master2 , we get CONNECTED When we run the zookeeper-client from master1 on the zookeeper server on master3 , we get CONNECTED Examples [root@master1 ~]# /usr/hdp/current/zookeeper-client/bin/zookeeper-client -server master1:2181 Connecting to master1:2181 Welcome to ZooKeeper! JLine support is enabled [zk: master1:2181(CONNECTING) 0] <-- we get CONNECTING instead to get CONNECTED [root@master1 ~]# /usr/hdp/current/zookeeper-client/bin/zookeeper-client -server master2:2181 Connecting to master2:2181 Welcome to ZooKeeper! JLine support is enabled WATCHER:: WatchedEvent state:SyncConnected type:None path:null [zk: master2:2181(CONNECTED) 0] [root@master1 ~]# /usr/hdp/current/zookeeper-client/bin/zookeeper-client -server master3:2181 Connecting to master3:2181 Welcome to ZooKeeper! JLine support is enabled WATCHER:: WatchedEvent state:SyncConnected type:None path:null [zk: master3:2181(CONNECTED) 0] so problem is only on master1 machine , and actually client cant connected to the zookeeper server on machine - master1 What could be the reason for that? more /etc/zookeeper/2.6.4.0-91/0/zoo.cfg clientPort=2181 syncLimit=15 autopurge.purgeInterval=24 maxClientCnxns=60 dataDir=/var/hadoop/zookeeper initLimit=30 tickTime=2000 autopurge.snapRetainCount=30 server.1=master1.sys89.com:2888:3888 server.2=master2.sys89.com:2888:3888 server.3=master3.sys89.com:2888:3888 cat /usr/hdp/current/zookeeper-client/bin/zookeeper-client #!/bin/bash export ZOOKEEPER_HOME=/usr/hdp/2.6.4.0-91//zookeeper export ZOOKEEPER_CONF=${ZOOKEEPER_HOME}/conf export CLASSPATH=$CLASSPATH:$ZOOKEEPER_CONF:$ZOOKEEPER_HOME/*:$ZOOKEEPER_HOME/lib/* export ZOOCFGDIR=${ZOOCFGDIR:-$ZOOKEEPER_CONF} env CLASSPATH=$CLASSPATH ${ZOOKEEPER_HOME}/bin/zkCli.sh "$@" we check the port 2181 and we get ok status telnet localhost 2181 Trying ::1 Connected to localhost. Escape character is '^]'. ... View more

Environment: HDP 2.6.4 Ambari – 2.6.1 3 zookeeper server hi all, In the first zookeeper server it seems that even after closing the connection to zookeeper is not getting closed, which causes the maximum number of client connections to be reached from a host - we have maxClientCnxns as 60 in zookeeper config As a result when a new application comes and tries to create a connection it fails. Example when Connections are: echo stat | nc 23.1.35.185 2181 Latency min/avg/max: 0/71/399 Received: 3031 Sent: 2407 Connections: 67 Outstanding: 622 Zxid: 0x130000004d Mode: follower Node count: 3730 But after some time when connection comes to ~70 we see echo stat | nc 23.1.35.185 2181 Ncat: Connection reset by peer. And We can see also many CLOSE_WAIT java 58936 zookeeper 60u IPv6 381963738 0t0 TCP zookeper_server.sys54.com:eforward->zookeper_server.sys54.com:44983 (CLOSE_WAIT) java 58936 zookeeper 61u IPv6 381963798 0t0 TCP zookeper_server.sys54.com:eforward->zookeper_server.sys54.com:45034 (CLOSE_WAIT) java 58936 zookeeper 62u IPv6 381963667 0t0 TCP zookeper_server.sys54.com:eforward->zookeper_server.sys54.com:44956 (CLOSE_WAIT) java 58936 zookeeper 63u IPv6 381949363 0t0 TCP zookeper_server.sys54.com:eforward->zookeper_server.sys54.com:44911 (CLOSE_WAIT) java 58936 zookeeper 64u IPv6 381964358 0t0 TCP zookeper_server.sys54.com:eforward->zookeper_server.sys54.com:44957 (CLOSE_WAIT) java 58936 zookeeper 65u IPv6 381963638 0t0 TCP zookeper_server.sys54.com:eforward->zookeper_server.sys54.com:44940 (CLOSE_WAIT) java 58936 zookeeper 66u IPv6 381963640 0t0 TCP zookeper_server.sys54.com:eforward->zookeper_server.sys54.com:44943 (CLOSE_WAIT) java 58936 zookeeper 67u IPv6 381963642 0t0 TCP zookeper_server.sys54.com:eforward->zookeper_server.sys54.com:44945 (CLOSE_WAIT) From the zookeeper log 2018-12-26 02:50:46,382 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /23.1.35.185 - max is 602018-12-26 02:50:46,429 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /23.1.35.197 - max is 602018-12-26 02:50:46,849 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /23.1.35.187 - max is 602018-12-26 02:50:47,645 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /23.1.35.197 - max is 602018-12-26 02:50:47,845 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /23.1.35.185 - max is 602018-12-26 02:50:48,180 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /23.1.35.187 - max is 602018-12-26 02:50:49,035 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /23.1.35.185 - max is 602018-12-26 02:50:49,375 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /23.1.35.187 - max is 60 In the ambari we can see also Connection failed: [Errno 104] Connection reset by peer to zookeper_server.sys54.com.:2181 I must to say that this not happening on zookeeper servers 2 and 3 so any hint why the connection are CLOSE_WAIT ? NOTE - if we increase the maxClientCnxns to 300 , its not help because after some time we get more the 300 connections and then we see from the log 2018-12-26 02:50:49,375 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too many connections from /23.1.35.187 - max is 60 ... View more