Zookeeper Issue

[#1]

Canary test of client connection to ZooKeeper and execution of basic operations succeeded though a session could not be established with one or more servers. Need help to know what to do in such situation?

[#2 · Accepted solution]

Check the ZooKeeper "Maximum Client Connections" (maxClientCnxns) property.
This often defaults to 60 and should be raised to 300.

This value sets a per-host limit on connections, so if there are more
connections to the ZooKeeper from processes on an individual host it will
begin rejecting connections.

The canary test is to connect to the ZooKeeper, create a znode, and delete
the znode.

[#3]

Hi,

I am new the CDH so, can you please explain me in details, how can I change it from 60 to 300 ? 

[#4]

Hi,

Also, when I check logs, I see the following error, can you help me in fixing this ?

Session 0x0 for server xxxxxxxxxxxxxx/10.4.2.110:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

[#5]

Hi,

I was able to change it from 60 to 300, now it seems working, the warning disppeared....

Thanks for helping Denloe.......you guys are great!!!

[#6]

Hi

We are using the zookeeper server on our google compute engine VMs. When we go to Cloudera manager, the status of Zookeeper says

that "ZooKeeper service zookeeper must have an odd number of servers." Obviously we thought of shutting down one of the servers.

Even when we stopper Zookeeper server on one of our nodes, the warning was still there. After this I have tried stopping, restarting and re-initializing the server but the warning message remains. Does anyone have any insight into this problem?

To make things worse, after I shut down the chosen node, I started getting this health warning message about that node.  Anyway, when I try restarting the zookeeper server on that machine, I get the following error message: "

Command Details: Restart this ServerLast Refreshed: Oct 29, 2015 10:23:37 PM UTC

Command Context Status Started at Ended at

Restart this Server	Server, hadp-inv-ibi-w3	Finished	Oct 29, 2015 10:23:24 PM UTC	Oct 29, 2015 10:23:36 PM UTC
Command (2133) has failed

Child Commands

All   Failed Only   Active Only

Command (Child commands) Context Status Started at Ended at

Start this Server	Server, hadp-inv-ibi-w3	Finished	Oct 29, 2015 10:23:24 PM UTC	Oct 29, 2015 10:23:36 PM UTC
Supervisor returned FATAL. Please check the role log file, stderr, or stdout. Program: zookeeper/zkserver.sh ["1","/var/lib/zookeeper"] Recent Log Entries Links to full logs: Stderr Stdout Role Log Details

Any help would be hugely appreciated.

Thanks

[#7]

The message implies you must have 1,3,5,7,... ZooKeeper servers in the
cluster. It doesn't matter if they're running or not at the time. How many
ZooKeeper servers do you have within the service?

[#8]

Hi

we have 4 zookeeper servers in the cluster. One of my hypotheses was stopping the zookeeper server on one of the 4 servers will solve the problem.

Since that's not working, what else can we do? Should we try uninstalling zookeeper on one of the nodes?...If so, could you please tell me how to do

that? I have been searching for a couple of days now for resources that would direct me to that direction. so far no luck.

if you need further info on our cluster, here's a screenshot for it: https://dl.dropboxusercontent.com/u/133690147/Capture.PNG

when i try to restart the w3 server (the server at the bottom of the picture above), I get the following error: https://dl.dropboxusercontent.com/u/133690147/Capture2.PNG

Thank you

[#9]

4 is not an ideal number. I'd suggest you simply stop and delete the "down"
ZooKeeper server from Cloudera Manager. It will remove the server from the
ensemble, you'd have to restart all of ZooKeeper to enforce the change.

[#10]

when you say "remove the server from the ensemble", do you mean delete it? Like selecting the delete option? : https://dl.dropboxusercontent.com/u/133690147/Capture3.PNG