Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Zookeeper Issue

avatar
Expert Contributor

Canary test of client connection to ZooKeeper and execution of basic operations succeeded though a session could not be established with one or more servers. Need help to know what to do in such situation?

1 ACCEPTED SOLUTION

avatar
Community Manager
Check the ZooKeeper "Maximum Client Connections" (maxClientCnxns) property.
This often defaults to 60 and should be raised to 300.

This value sets a per-host limit on connections, so if there are more
connections to the ZooKeeper from processes on an individual host it will
begin rejecting connections.

The canary test is to connect to the ZooKeeper, create a znode, and delete
the znode.



David Wilder, Community Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

View solution in original post

10 REPLIES 10

avatar
Community Manager
Check the ZooKeeper "Maximum Client Connections" (maxClientCnxns) property.
This often defaults to 60 and should be raised to 300.

This value sets a per-host limit on connections, so if there are more
connections to the ZooKeeper from processes on an individual host it will
begin rejecting connections.

The canary test is to connect to the ZooKeeper, create a znode, and delete
the znode.



David Wilder, Community Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

avatar
Expert Contributor

Hi,

 

I am new the CDH so, can you please explain me in details, how can I change it from 60 to 300 ? 

avatar
Expert Contributor

Hi,

 

Also, when I check logs, I see the following error, can you help me in fixing this ?

 

Session 0x0 for server xxxxxxxxxxxxxx/10.4.2.110:2181, unexpected error, closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

avatar
Expert Contributor

Hi,

 

I was able to change it from 60 to 300, now it seems working, the warning disppeared....

 

Thanks for helping Denloe.......you guys are great!!!

avatar
New Contributor

Hi

 

We are using the zookeeper server on our google compute engine VMs. When we go to Cloudera manager, the status of Zookeeper says

that "ZooKeeper service zookeeper must have an odd number of servers." Obviously we thought of shutting down one of the servers.

Even when we stopper Zookeeper server on one of our nodes, the warning was still there. After this I have tried stopping, restarting and re-initializing the server but the warning message remains. Does anyone have any insight into this problem?

 

To make things worse, after I shut down the chosen node, I started getting this health warning message about that node.  Anyway, when I try restarting the zookeeper server on that machine, I get the following error message: "

avatar
The message implies you must have 1,3,5,7,... ZooKeeper servers in the
cluster. It doesn't matter if they're running or not at the time. How many
ZooKeeper servers do you have within the service?

Regards,
Gautam Gopalakrishnan

avatar
New Contributor

Hi

 

we have 4 zookeeper servers in the cluster. One of my hypotheses was stopping the zookeeper server on one of the 4 servers will solve the problem.

Since that's not working, what else can we do? Should we try uninstalling zookeeper on one of the nodes?...If so, could you please tell me how to do

that? I have been searching for a couple of days now for resources that would direct me to that direction. so far no luck.

 

if you need further info on our cluster, here's a screenshot for it: https://dl.dropboxusercontent.com/u/133690147/Capture.PNG

 

when i try to restart the w3 server (the server at the bottom of the picture above), I get the following error: https://dl.dropboxusercontent.com/u/133690147/Capture2.PNG

 

Thank you

avatar
4 is not an ideal number. I'd suggest you simply stop and delete the "down"
ZooKeeper server from Cloudera Manager. It will remove the server from the
ensemble, you'd have to restart all of ZooKeeper to enforce the change.

Regards,
Gautam Gopalakrishnan

avatar
New Contributor

when you say "remove the server from the ensemble", do you mean delete it? Like selecting the delete option? : https://dl.dropboxusercontent.com/u/133690147/Capture3.PNG