zookeeper-client cannot connect to zookeeper server

[#1]

hi all

In my HDP cluster, I install 3 zookeeper-servers and zookeeper client on 3 nodes ( master1 , master2 , master3 ) ,.

all nodes are on redhat machine version 7.2

When we run the zookeeper-client from master1 on the zookeeper server on master1 , we get CONNECTING

When we run the zookeeper-client from master1 on the zookeeper server on master2 , we get CONNECTED

When we run the zookeeper-client from master1 on the zookeeper server on master3 , we get CONNECTED

Examples

[root@master1 ~]# /usr/hdp/current/zookeeper-client/bin/zookeeper-client -server master1:2181
Connecting to master1:2181
Welcome to ZooKeeper!
JLine support is enabled
[zk: master1:2181(CONNECTING) 0]        <-- we get CONNECTING instead to get CONNECTED


[root@master1 ~]# /usr/hdp/current/zookeeper-client/bin/zookeeper-client -server master2:2181
Connecting to master2:2181
Welcome to ZooKeeper!
JLine support is enabled
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: master2:2181(CONNECTED) 0] 


[root@master1 ~]# /usr/hdp/current/zookeeper-client/bin/zookeeper-client -server master3:2181
Connecting to master3:2181
Welcome to ZooKeeper!
JLine support is enabled
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: master3:2181(CONNECTED) 0] 

so problem is only on master1 machine , and actually client cant connected to the zookeeper server on machine - master1

What could be the reason for that?

more /etc/zookeeper/2.6.4.0-91/0/zoo.cfg
clientPort=2181
syncLimit=15
autopurge.purgeInterval=24
maxClientCnxns=60
dataDir=/var/hadoop/zookeeper
initLimit=30
tickTime=2000
autopurge.snapRetainCount=30
server.1=master1.sys89.com:2888:3888
server.2=master2.sys89.com:2888:3888
server.3=master3.sys89.com:2888:3888

cat /usr/hdp/current/zookeeper-client/bin/zookeeper-client
#!/bin/bash


export ZOOKEEPER_HOME=/usr/hdp/2.6.4.0-91//zookeeper
export ZOOKEEPER_CONF=${ZOOKEEPER_HOME}/conf
export CLASSPATH=$CLASSPATH:$ZOOKEEPER_CONF:$ZOOKEEPER_HOME/*:$ZOOKEEPER_HOME/lib/*
export ZOOCFGDIR=${ZOOCFGDIR:-$ZOOKEEPER_CONF}
env CLASSPATH=$CLASSPATH ${ZOOKEEPER_HOME}/bin/zkCli.sh "$@"


we check the port 2181 and we get ok status

telnet localhost 2181
Trying ::1
Connected to localhost.
Escape character is '^]'.

[#2]

@Michael Bronson

Please can you check the zookeeper logs (/var/log/zookeeper) of master1.sys89.com. This can happen if there are too many open connections. Check where there were any warning messages stating with “Too many connections from {IP address of master1.sys89.com}”. Using netstat command also you can verify

netstat -no | grep :2181 | wc -l

To fix this issue, kindly clear up all stale connections manually or try increasing the maxClientCnxns setting at /etc/zookeeper/2.6.4.0-91/0/zoo.cfg. From your zoo.cfg file I can see value is maxClientCnxns=60 which is default. You can increase it by adding the maxClientCnxns=4096 and restart respective affected services.

[#3]

we already do all this steps , if we increase the axClientCnxns=500 or to 5000, then after some time all ports are in used and we get CLOSE_WAIT so increasing it isn't solution , and from the log we not see hint about - why zookper clinet not connect zookper server

[#4]

by the way , if we restart the zookeper , then open connections are less then ,maxClientCnxns and we still get CONNECTING , this mean zookeper client cant connect to zookeper server

[#5]

@Michael Bronson

Isn't this a duplicate thread for the same problem

https://community.hortonworks.com/questions/232197/too-many-connections-on-zookeper-server.html?chil...

As the problem is only on Master1 can you check that your /etc/hosts entries are correct? and FW is not active on the master1. your zoo.cfg looks correct.
Last resort would be to delete and install the zookeeper service.

[#6]

@Geoffrey Shelton Okot , yes this is the same problem , but with different aspect , we checked the /etc/hosts , and file is OK , FW is disable

anyway , we can use IP instead localhost as the following:

/usr/hdp/current/zookeeper-client/bin/zookeeper-client -server SOME_IP:2181

[#7]

@Michael Bronson
So if you use the IP does it work if so please check the DNS resolution for this particular host.

[#8]

no even we use IP its not works

[#9]

@Michael Bronson
Can you share the hosts' entries on the 3 Zk nodes ?

[#10]

[root@master1 hdp]# more /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

[root@master2 more /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6


[root@master3 ~]# more /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

[#11]

Geoffrey as you can see the hosts seems to be fine

[#12]

@Michael Bronson

I don't see the hostname entries of master1-->Master3 in the below format in your host file?

IP:FQDN:ALIAS

Did you intentionally remove them? Can you validate your DNS is functioning

Find out the IP address of Domain

$ nsloookup FQDN 

Find out Reverse Domain Lookup

$ nslookup IP

[#13]

@Michael Bronson

This is what I was trying to say, below is my single node HDP and AD server.

In your case, unless you have a DNS server you should have 3 entries for master1,master2 and master3

[#14]

@Geoffrey Shelton Okot we are using DNS serve , so all entries of hostnames and ip's in the DNS server

[#15]

yes we have resolving of all master1/2/3 names and also ip's , we checked that with host command

[#16]

hi Geoffrey , I post anew thread - https://community.hortonworks.com/questions/231177/metrics-failed-on-orgapachehadoophbasezookeeperzo... , but I see this post not appears in the hortonworks questions ,could you help me to understand why ?