zookeper + Client session timed out, have not heard from server

[#1]

hi all

from the first zookeeper server , we are trying to do ( from zookeeper session )

ls /

but we got error about

Client session timed out, have not heard from server in 15005ms for sessionid 0x0, closing socket connection and attempting reconnect

what could be the problem here?

full example from our zookeeper server machine

[zk: localhost:2181(CONNECTING) 0] ls /

2018-10-16 14:03:22,585 - INFO  [main-SendThread(localhost:2181):ClientCnxn$SendThread@1140] - Client session timed out, have not heard from server in 15005ms for sessionid 0x0, closing socket connection and attempting reconnect
Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)
        at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1500)
        at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:723)
        at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:591)
        at org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:363)
        at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:323)
        at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:282)
You have mail in /var/spool/mail/root

[#2]

@Michael Bronson,

From the logs it looks like the client is not yet connected to the server

[zk: localhost:2181(CONNECTING)0]

If it is connected , you should get CONNECTED instead of CONNECTING. If your cluster is Kerberized, you need to run kinit before connecting to the zookeeper client. You can run the below steps

# kinit -kt /etc/security/keytabs/hdfs.headless.keytab {principal}
# zookeeper-client -server {zk-host}:2181
## zk: zkhost-1(CONNECTED) 1] ls /

[#3]

we not have the command on the zookeper server - ( kinit )

[#4]

@Michael Bronson,

Is your cluster Kerberized? If it is not kerberized you may not have kinit installed. In that case, you can just run these commands

# su hdfs
# zookeeper-client -server {zk-host}:2181
## zk: zkhost-1(CONNECTED) 1] ls /

If this doesn't work, try restarting zookeeper server and try again.

[#5]

we restart the zookeper server , but after that we try again but with the same isshue

[#6]

@Michael Bronson,

Are you running the command from the same node where zookeeper is running ? Can you please paste the command that you are running. Can you try passing proper hostname instead of localhost:2181 while running the command?

[#7]

this is the command that we are run

master01 - is the zookeper server 
/usr/hdp/2.6.0.3-8/zookeeper/bin/zookeeper-client -server master01:2181

[#8]

this is also happens when we run

/usr/hdp/2.6.0.3-8/zookeeper/bin/zookeeper-client -server localhost:2181

[#9]

yes we run from the same node