Apache ZooKeeper is a “high-performance coordination service
for distributed applications.” Most users do not use ZooKeeper directly; however,
most users are also hard-pressed to deploy a Hadoop-based architecture that
doesn’t rely on ZooKeeper in some way. With its prevalence in the data-center, resource
management within ZooKeeper is paramount to ensure that the various
applications and services relying on ZooKeeper are able to access it in a
timely manner. To this end, one of ZooKeeper’s protection mechanisms is known
as “max client connections” or “
maxClientCnxns”.
maxClientCnxns refers to a configuration property that can
be added to the zoo.cfg configuration file. This property limits the number of
active connections from a host, specified by IP address, to a single ZooKeeper
server. By default, this limit is 60 active connections; one host is not
allowed to have more than 60 active connections open to one ZooKeeper server.
Changes to this property in the zoo.cfg file require a restart of ZooKeeper.
This is a simple way that ZooKeeper prevents clients from performing a denial
of service attack against ZooKeeper (maliciously or unwittingly) as well as
limiting the amount of memory required by these client connections.
The reason this property is so important is that it can
effectively deny all access from a host inside of a cluster to a ZooKeeper
server. This can have a severe performance and stability impacts on a cluster.
For example, if a node running an Apache HBase RegionServer hits the
maxClientCnxns limit, all future requests made by that RegionServer to that
ZooKeeper server would be dropped until the overall number of connections to
the ZooKeeper server are reduced. Perhaps the worst part about this is that
processes other than HBase running on the same node (e.g. YARN containers as a
part of a MapReduce job) could also eat into the allowed connections from the
same host.
On a positive note, it is simple to recognize when this rate
limiting is happening and also simple to determine the problematic clients on
the rate-limited host. First, there is a very clear error message in the
ZooKeeper server log which identifies the host being rate-limited and the
current active connections limit:
“Too many connections from 10.0.0.1 – max is 60”
This error message is stating that a client from the host
with IP address 10.0.0.1 is trying to connect to this ZooKeeper server, but the
limit is 60 connections. As such, the current connection will be dropped. At
this point, we know the host where these connections are coming from, but we
don’t know what applications on that host are making them. We can use a network
analysis tool such as `netstat` to determine the applications on the client host,
in this case 10.0.0.1 (let’s assume our ZooKeeper server is on 10.0.0.5):
netstat -nape | awk ‘{if ($5 == “10.0.0.5:2181”) print $4, $9;}’
This command will list the local address and process
identifier for each connection, only where the remote address is our ZooKeeper
server and the ZooKeeper service port (2181). Similarly, we can further group
this data to give us a count of outgoing connections by process identifier to
the ZooKeeper server.
netstat -nape | awk ‘{if ($5 == “10.0.0.5:2181”) print $9;}’ | sort | uniq –c
This command will report a count of connections to the
ZooKeeper server. This can be extremely helpful in identifying misbehaving
applications causing issues. Additionally, we can use some of the “four letter
word” commands to further give us information about the active connections to a
ZooKeeper server. Using netcat, either of the following could be used:
echo “stat” | nc 10.0.0.5 2181
echo “cons” | nc 10.0.0.5 2181
Each of these commands will output data which contains information about the active connections to the given ZooKeeper server.
To summarize, the maxClientCnxns property in zoo.cfg is used
by the ZooKeeper server to limit incoming connections to the ZooKeeper from a
single host. By default, this limit is 60. When this limit is reached, new
connections to the ZooKeeper server from the given host will be immediately
dropped. This rate-limiting can be observed in the ZooKeeper log and offending
applications can be identified by using network tools like netstat. Changes to
maxClientCnxns must be accompanied with a restart of the ZooKeeper server.
ZooKeeper configuration property documentation
ZooKeeper four letter words documentation
... View more
