zookeeper-client cannot connect to zookeeper server

[#1]

hi all

In my HDP cluster, I install 3 zookeeper-servers and zookeeper client on 3 nodes ( master1 , master2 , master3 ) ,.

all nodes are on redhat machine version 7.2

When we run the zookeeper-client from master1 on the zookeeper server on master1 , we get CONNECTING

When we run the zookeeper-client from master1 on the zookeeper server on master2 , we get CONNECTED

When we run the zookeeper-client from master1 on the zookeeper server on master3 , we get CONNECTED

Examples

[root@master1 ~]# /usr/hdp/current/zookeeper-client/bin/zookeeper-client -server master1:2181
Connecting to master1:2181
Welcome to ZooKeeper!
JLine support is enabled
[zk: master1:2181(CONNECTING) 0]        <-- we get CONNECTING instead to get CONNECTED


[root@master1 ~]# /usr/hdp/current/zookeeper-client/bin/zookeeper-client -server master2:2181
Connecting to master2:2181
Welcome to ZooKeeper!
JLine support is enabled
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: master2:2181(CONNECTED) 0] 


[root@master1 ~]# /usr/hdp/current/zookeeper-client/bin/zookeeper-client -server master3:2181
Connecting to master3:2181
Welcome to ZooKeeper!
JLine support is enabled
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: master3:2181(CONNECTED) 0] 

so problem is only on master1 machine , and actually client cant connected to the zookeeper server on machine - master1

What could be the reason for that?

more /etc/zookeeper/2.6.4.0-91/0/zoo.cfg
clientPort=2181
syncLimit=15
autopurge.purgeInterval=24
maxClientCnxns=60
dataDir=/var/hadoop/zookeeper
initLimit=30
tickTime=2000
autopurge.snapRetainCount=30
server.1=master1.sys89.com:2888:3888
server.2=master2.sys89.com:2888:3888
server.3=master3.sys89.com:2888:3888

cat /usr/hdp/current/zookeeper-client/bin/zookeeper-client
#!/bin/bash


export ZOOKEEPER_HOME=/usr/hdp/2.6.4.0-91//zookeeper
export ZOOKEEPER_CONF=${ZOOKEEPER_HOME}/conf
export CLASSPATH=$CLASSPATH:$ZOOKEEPER_CONF:$ZOOKEEPER_HOME/*:$ZOOKEEPER_HOME/lib/*
export ZOOCFGDIR=${ZOOCFGDIR:-$ZOOKEEPER_CONF}
env CLASSPATH=$CLASSPATH ${ZOOKEEPER_HOME}/bin/zkCli.sh "$@"


we check the port 2181 and we get ok status

telnet localhost 2181
Trying ::1
Connected to localhost.
Escape character is '^]'.

[#2]

Geoffrey as you can see the hosts seems to be fine

[#3]

@Michael Bronson

I don't see the hostname entries of master1-->Master3 in the below format in your host file?

IP:FQDN:ALIAS

Did you intentionally remove them? Can you validate your DNS is functioning

Find out the IP address of Domain

$ nsloookup FQDN 

Find out Reverse Domain Lookup

$ nslookup IP

[#4]

@Michael Bronson

This is what I was trying to say, below is my single node HDP and AD server.

In your case, unless you have a DNS server you should have 3 entries for master1,master2 and master3

[#5]

@Geoffrey Shelton Okot we are using DNS serve , so all entries of hostnames and ip's in the DNS server

[#6]

yes we have resolving of all master1/2/3 names and also ip's , we checked that with host command

[#7]

hi Geoffrey , I post anew thread - https://community.hortonworks.com/questions/231177/metrics-failed-on-orgapachehadoophbasezookeeperzo... , but I see this post not appears in the hortonworks questions ,could you help me to understand why ?