ZooKeeper Client Configuration: Connecting to Remote Quorum vs. Localhost

[#1]

In my HDP cluster, I install 3 zookeeper-servers on 3 nodes, other nodes I just only install zookeeper-client.

When I stand on Zookeeper-Server nodes, I can run zookeeper-client command OK.

But when I stand on Zookeeper-Client nodes, I run zookeeper-client and can not connect to zookeeper servers

2017-03-01 23:19:57,076 - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@23ab930d
Welcome to ZooKeeper!
2017-03-01 23:19:57,112 - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1019] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2017-03-01 23:19:57,205 - WARN  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1146] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1125)
[zk: localhost:2181(CONNECTING) 0] 2017-03-01 23:19:58,312 - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1019] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-03-01 23:19:58,314 - WARN  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1146] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1125)
2017-03-01 23:19:59,415 - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1019] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-03-01 23:19:59,416 - WARN  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1146] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1125)
2017-03-01 23:20:00,517 - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1019] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2017-03-01 23:20:00,518 - WARN  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1146] - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1125)

So anyone can have solution for this problem

[#2 · Accepted solution]

by default it is connecting to localhost:2181, since there is no zookeeper server running on client host so it fails,

please specify zookeeper server and port in your command then it will work. eg:

/usr/hdp/current/zookeeper-client/bin/zookeeper-client -server <ZK1>:2181,<ZK2>:2181

[#3]

@Deepak Sharma

Thank you, that command is worked.

I see in other node that I don't install ZK server and ZK client. I can use use command without any warning

/usr/hdp/current/zookeeper-client/bin/zookeeper-client

So Do you know why it happen ?

[#4]

it happens because zookeeper-client command is connecting to the localhost so whenever you run it without specifying server on the node where zk server is installed , it works because localhost has zookeeper. but when you run it on other node then it does not work since there is no zookeeper installed on the localhost

[#5]

@Deepak Sharma

in the command "/usr/hdp/current/zookeeper-client/bin/zookeeper-client -server <ZK1>:2181,<ZK2>:2181"

<ZK1>:2181,<ZK2>:2181 ---- I dint understand. Could you explain it.

[#6]

@Hoang Le

Can you please confirm in the following file whatever host/port are mentioned those are accessible from the "Zookeeper Client" machine?

/etc/zookeeper/conf/zoo.cfg 

.

This file should show the ZK servers details that should be reachable properly from client. So please check the host/port details and try doing a "telnet" to verify if those are reachable or not?

clientPort=2181
initLimit=10
autopurge.purgeInterval=24
syncLimit=5
tickTime=2000
dataDir=/hadoop/zookeeper
autopurge.snapRetainCount=30
server.1=erie1.example.com:2888:3888
server.2=erie2.example.com:2888:3888
server.3=erie3.example.com:2888:3888
server.4=erie4.example.com:2888:3888

.

[#7]

@Jay SenSharma

/usr/hdp/current/zookeeper-client/bin/zookeeper-client -server <ZK1>:2181,<ZK2>:2181 that command work with all ZK server

[#8]

@Hoang Le

The command "/usr/hdp/current/zookeeper-client/bin/zookeeper-client" should work becase it is able to locate the zookeeper conf directory as following:

# cat /usr/hdp/current/zookeeper-client/bin/zookeeper-client 
#!/bin/bash

export ZOOKEEPER_HOME=/usr/hdp/2.5.0.0-1245//zookeeper
export ZOOKEEPER_CONF=${ZOOKEEPER_HOME}/conf
export CLASSPATH=$CLASSPATH:$ZOOKEEPER_CONF:$ZOOKEEPER_HOME/*:$ZOOKEEPER_HOME/lib/*
export ZOOCFGDIR=${ZOOCFGDIR:-$ZOOKEEPER_CONF}
env CLASSPATH=$CLASSPATH ${ZOOKEEPER_HOME}/bin/zkCli.sh "$@"

.

So as long as it is able to locate the correct "ZOOKEEPER_CONF" it should be able to connect to the local zookeeper.