why zookeeper goes down when maximum client connection reached 60

[#1]

Hi, We upgraded our prod cluster from HDP 2.3 to HDP 2..5 and after few days we have been getting new errors which never got in our former cluster. suddenly our zookeeper sever went down then we saw log "maximum client connection reached 60" then we changed the attribute to 200 in zoo.cfg. Again that host reached the max connection that is 200. So I dont understand why its hitting maximum connection everytime. Because once application job completes then session should be closed , why clint connections are active...Please advice me what should be done....T

[#2]

Hi @Artem Ervits can you help.

[#3]

@Amit Panda thanks for reaching out, it is hard to say what the issue is. Can you post the logs from zookeeper to see? Also, can you publish the difference in config changes between before upgrade and now by looking at Ambari config history and comparing the two? Even a screenshot will do there. Btw, make sure logs are clear of any site specific information as this is a public forum.

[#4]

Hi @Artem Ervits Thanks for response. I checked /etc/zookeeper/conf/zoo.cfg and after upgradation I found same values as before. So I could not find out the exact problem on looking into the configuration files.

2017-02-08 00:41:52,616 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x15a19e6cf730013, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:745) 2017-02-08 00:41:52,675 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /172.17.0.2:34852 which had sessionid 0x15a19e6cf730013 2017-02-08 00:41:54,441 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /172.17.0.2:40119 2017-02-08 00:41:54,442 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861] - Client attempting to renew session 0x15a19e6cf730013 at /172.17.0.2:40119 2017-02-08 00:41:54,443 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@617] - Established session 0x15a19e6cf730013 with negotiated timeout 10000 for client /172.17.0.2:40119 2017-02-08 00:42:19,328 - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x15a19e6cf730013, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:745) 2017-02-08 00:42:19,329 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /172.17.0.2:40119 which had sessionid 0x15a19e6cf730013 2017-02-08 00:42:20,004 - INFO [SessionTracker:ZooKeeperServer@347] - Expiring session 0x15a19e6cf730013, timeout of 10000ms exceeded 2017-02-08 00:42:20,005 - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor@494] - Processed session termination for sessionid: 0x15a19e6cf730013 2017-02-08 00:42:20,477 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /172.17.0.2:40193 2017-02-08 00:42:20,488 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861] - Client attempting to renew session 0x15a19e6cf730013 at /172.17.0.2:40193 2017-02-08 00:42:20,489 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@610] - Invalid session 0x15a19e6cf730013 for client /172.17.0.2:40193, probably expired 2017-02-08 00:42:20,489 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /172.17.0.2:40193 which had sessionid 0x15a19e6cf730013 2017-02-08 00:42:20,588 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /172.17.0.2:40194 2017-02-08 00:42:20,589 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to establish new session at /172.17.0.2:40194 2017-02-08 00:42:20,626 - INFO [SyncThread:0:ZooKeeperServer@617] - Established session 0x15a19e6cf730015 with negotiated timeout 10000 for client /172.17.0.2:40194

Above are logs but I could not find any issue from here.

Please advise .

[#5]

@Amit Panda I can't tell from this log, you should consider opening a ticket with support and run a SmartSense analysis on the configuration.

[#6]

Thanks a lot..Let me try that

[#7]

@Amit Panda it will help community if you publish the results of your findings with support. It is possible it will serve as reference for other customers having same issue.