question Re: Zookeeper Issue in Archives of Support Questions (Read Only)

Zookeeper Issue

Jais — Fri, 16 Sep 2022 09:24:49 GMT

Canary test of client connection to ZooKeeper and execution of basic operations succeeded though a session could not be established with one or more servers. Need help to know what to do in such situation?

Re: Zookeeper Issue

denloe — Tue, 17 Mar 2015 17:05:34 GMT

Check the ZooKeeper "Maximum Client Connections" (maxClientCnxns) property.
This often defaults to 60 and should be raised to 300.

This value sets a per-host limit on connections, so if there are more
connections to the ZooKeeper from processes on an individual host it will
begin rejecting connections.

The canary test is to connect to the ZooKeeper, create a znode, and delete
the znode.

Re: Zookeeper Issue

Jais — Tue, 17 Mar 2015 17:07:19 GMT

Hi,

I am new the CDH so, can you please explain me in details, how can I change it from 60 to 300 ?

Re: Zookeeper Issue

Jais — Tue, 17 Mar 2015 17:12:00 GMT

Hi,

Also, when I check logs, I see the following error, can you help me in fixing this ?

Session 0x0 for server xxxxxxxxxxxxxx/10.4.2.110:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

Re: Zookeeper Issue

Jais — Tue, 17 Mar 2015 17:16:57 GMT

Hi,

I was able to change it from 60 to 300, now it seems working, the warning disppeared....

Thanks for helping Denloe.......you guys are great!!!

Zookeeper Warning: "ZooKeeper service zookeeper must have an odd number of servers."

muntasir — Fri, 30 Oct 2015 12:18:20 GMT

We are using the zookeeper server on our google compute engine VMs. When we go to Cloudera manager, the status of Zookeeper says

that "ZooKeeper service zookeeper must have an odd number of servers." Obviously we thought of shutting down one of the servers.

Even when we stopper Zookeeper server on one of our nodes, the warning was still there. After this I have tried stopping, restarting and re-initializing the server but the warning message remains. Does anyone have any insight into this problem?

To make things worse, after I shut down the chosen node, I started getting this health warning message about that node. Anyway, when I try restarting the zookeeper server on that machine, I get the following error message: "

Re: Zookeeper Warning: "ZooKeeper service zookeeper must have an odd number of servers."

GautamG — Thu, 29 Oct 2015 23:05:56 GMT

The message implies you must have 1,3,5,7,... ZooKeeper servers in the
cluster. It doesn't matter if they're running or not at the time. How many
ZooKeeper servers do you have within the service?

Re: Zookeeper Warning: "ZooKeeper service zookeeper must have an odd number of servers."

muntasir — Thu, 29 Oct 2015 23:34:15 GMT

we have 4 zookeeper servers in the cluster. One of my hypotheses was stopping the zookeeper server on one of the 4 servers will solve the problem.

Since that's not working, what else can we do? Should we try uninstalling zookeeper on one of the nodes?...If so, could you please tell me how to do

that? I have been searching for a couple of days now for resources that would direct me to that direction. so far no luck.

if you need further info on our cluster, here's a screenshot for it: https://dl.dropboxusercontent.com/u/133690147/Capture.PNG

when i try to restart the w3 server (the server at the bottom of the picture above), I get the following error: https://dl.dropboxusercontent.com/u/133690147/Capture2.PNG

Thank you

Re: Zookeeper Warning: "ZooKeeper service zookeeper must have an odd number of servers."

GautamG — Thu, 29 Oct 2015 23:36:56 GMT

4 is not an ideal number. I'd suggest you simply stop and delete the "down"
ZooKeeper server from Cloudera Manager. It will remove the server from the
ensemble, you'd have to restart all of ZooKeeper to enforce the change.

Re: Zookeeper Warning: "ZooKeeper service zookeeper must have an odd number of servers."

muntasir — Thu, 29 Oct 2015 23:52:14 GMT

when you say "remove the server from the ensemble", do you mean delete it? Like selecting the delete option? : https://dl.dropboxusercontent.com/u/133690147/Capture3.PNG

Re: Zookeeper Warning: "ZooKeeper service zookeeper must have an odd number of servers."

GautamG — Thu, 29 Oct 2015 23:53:56 GMT

Yes, by deleting it from the list, Cloudera Manager will automatically
remove it from the ensemble and update the config files of the other three
servers. They would need a restart to see the change.

Restart this Server	Server, hadp-inv-ibi-w3	Finished	Oct 29, 2015 10:23:24 PM UTC	Oct 29, 2015 10:23:36 PM UTC
Command (2133) has failed