Understanding Apache ZooKeeper Connection Rate Limiting

[#1]

Apache ZooKeeper is a “high-performance coordination service for distributed applications.” Most users do not use ZooKeeper directly; however, most users are also hard-pressed to deploy a Hadoop-based architecture that doesn’t rely on ZooKeeper in some way. With its prevalence in the data-center, resource management within ZooKeeper is paramount to ensure that the various applications and services relying on ZooKeeper are able to access it in a timely manner. To this end, one of ZooKeeper’s protection mechanisms is known as “max client connections” or “ maxClientCnxns”.

maxClientCnxns refers to a configuration property that can be added to the zoo.cfg configuration file. This property limits the number of active connections from a host, specified by IP address, to a single ZooKeeper server. By default, this limit is 60 active connections; one host is not allowed to have more than 60 active connections open to one ZooKeeper server. Changes to this property in the zoo.cfg file require a restart of ZooKeeper. This is a simple way that ZooKeeper prevents clients from performing a denial of service attack against ZooKeeper (maliciously or unwittingly) as well as limiting the amount of memory required by these client connections.

The reason this property is so important is that it can effectively deny all access from a host inside of a cluster to a ZooKeeper server. This can have a severe performance and stability impacts on a cluster. For example, if a node running an Apache HBase RegionServer hits the maxClientCnxns limit, all future requests made by that RegionServer to that ZooKeeper server would be dropped until the overall number of connections to the ZooKeeper server are reduced. Perhaps the worst part about this is that processes other than HBase running on the same node (e.g. YARN containers as a part of a MapReduce job) could also eat into the allowed connections from the same host.

On a positive note, it is simple to recognize when this rate limiting is happening and also simple to determine the problematic clients on the rate-limited host. First, there is a very clear error message in the ZooKeeper server log which identifies the host being rate-limited and the current active connections limit:

“Too many connections from 10.0.0.1 – max is 60”

This error message is stating that a client from the host with IP address 10.0.0.1 is trying to connect to this ZooKeeper server, but the limit is 60 connections. As such, the current connection will be dropped. At this point, we know the host where these connections are coming from, but we don’t know what applications on that host are making them. We can use a network analysis tool such as `netstat` to determine the applications on the client host, in this case 10.0.0.1 (let’s assume our ZooKeeper server is on 10.0.0.5):

netstat -nape | awk ‘{if ($5 == “10.0.0.5:2181”) print $4, $9;}’

This command will list the local address and process identifier for each connection, only where the remote address is our ZooKeeper server and the ZooKeeper service port (2181). Similarly, we can further group this data to give us a count of outgoing connections by process identifier to the ZooKeeper server.

netstat -nape | awk ‘{if ($5 == “10.0.0.5:2181”) print $9;}’ | sort | uniq –c

This command will report a count of connections to the ZooKeeper server. This can be extremely helpful in identifying misbehaving applications causing issues. Additionally, we can use some of the “four letter word” commands to further give us information about the active connections to a ZooKeeper server. Using netcat, either of the following could be used:

echo “stat” | nc 10.0.0.5 2181

echo “cons” | nc 10.0.0.5 2181

Each of these commands will output data which contains information about the active connections to the given ZooKeeper server.

To summarize, the maxClientCnxns property in zoo.cfg is used by the ZooKeeper server to limit incoming connections to the ZooKeeper from a single host. By default, this limit is 60. When this limit is reached, new connections to the ZooKeeper server from the given host will be immediately dropped. This rate-limiting can be observed in the ZooKeeper log and offending applications can be identified by using network tools like netstat. Changes to maxClientCnxns must be accompanied with a restart of the ZooKeeper server.

ZooKeeper configuration property documentation

ZooKeeper four letter words documentation