How to decide, how many zookeepers should I have?

[#1]

Guys,

We have configured three zookeeper instances for a cluster when the cluster was 8 node cluster. Now it has gone bigger and is 28 nodes, and may go up to 40 shortly. I am a bit confused here, should we add more ZK services or its ok to maintain three.

What are the implications of increasing the ZK Servers to 5 or 7 or 9 ? And what are the implications of keeping them to three?

Thanks.

[#2]

@Smart Solutions There are two aspects - cluster size and load as explained below.

1. 3 zookeepers on decent H/W machines for cluster size upto 300-500 nodes is fine assuming that you don't have Kafka and Storm too as having Kafka, Storm, HDFS, Hive, HBase, YARN all depend on one quorum of 3 zookeepers can be really heavy for one quorum.

So, in your case for 40 node cluster size, 3 zookeepers should be good. But if you have all these components and heavy lifting being done in Kafka/Storm as well, then go with 2 quorums of 3 Zookeepers each i.e. 1 quorum of 3ZKs for HDFS, YARN, HBase, Hive and other quorum of 3 ZKs for Storm and Kafka.

2. Now in terms of increasing number of Zookeeper Servers in one quorum, while it is recommended to keep 3 ZKs as minimal and you are good there but if your cluster size increases to 500+ nodes, you should increase number of ZKs in one quorum to 5 or 7 depending on cluster size. Organizations running large clusters i.e. 1000-2500 nodes have 5 to 7 Zookeeper Servers per quorum in production.

[#3]

Can I have 5 zookeeper nodes and 1 quorum for Nifi, Spark, Kafka, PostgreSQL, TimescaleDB?

[#4]

@kalhan 

While it is possible to have a single ZK cluster to support multiple services, It is the recommendation that NiFi have its own dedicated ZK cluster.  NiFi cluster stability is dependent on ZK and many of the NiFi processors that can be used depend on on Cluster state which is also stored in ZK.  IF ZK becomes overburdened it can affect overall stability and performance of NiFi.

If you found any of the answers provided on this query helped you, please select "accept solution" on each of them.

Thank you,

Matt

Hope this helps.

[#5]

Hi @Smart Solutions.

Generally for that level, 3 zookeepers should be fine. You could bump it up to 5 if you start seeing issues, but we rarely see clusters go higher than that for a zookeeper instance as much higher starts to create quorum overheads.

There's no simple rule of thumb for this, it's as much an art as it is a science, as it depends on the workloads and how chatty they are with your current zk's.

Rather than going to larger clusters of zk's people tend to split out certain services to their own zk's when they're putting more pressure on an otherwise fairly quiet zk cluster.

So. In short, even up to 40 nodes could be fine, but keep an eye on your zk response times etc, if you start to see issues, then maybe move to 5 zk nodes, or consider splitting out the heaviest service to it's own zk cluster.

Finally keep an eye on the I/O requirements, dedicate spindles to ZK quorum, don't make it share other busier disks.

Hope that helps!

[#6]

One more way to look at this: If you have 3 ZKs you can afford to lose one, if you have 5 you can afford to lose two. If your organization is agressively applying security patches and other upgrades, like firmware, kernel, Java, other packages used by Hadoop, and taking nodes down to do the job, then during those upgrades with 3 ZKs, you ZK runs with only two nodes, and if you are unlucky and one of them goes down, then your whole cluster will go down. So, in this case 5 are bettter. However, the more ZK nodes you have, the slower the ZK becomes for writes.

[#7]

For a single node nifi cluster, is 1 ZK host alright?

[#8]

@Cl0ck 

This should probably be its own question in the community...

What is the purpose of building a 1 node NiFi cluster?
NiFi clusters add overhead to support the zero master clustering.  
- Things like zookeeper, cluster coordinators, heartbeats....

If you only ever intend to have one NiFi instance, you should just set the "is.node" to false in the nifi.properties file so your install is standalone which has no dependency on zookeeper at all.

If you plan on adding additional nodes to your one node cluster, then setting up a one node cluster initially is ok.

To answer your question:
- 1 zookeeper node does not give you quorum.  Zookeeper should be set up using an odd number of nodes (3 or 5 nodes recommended).  Without quorum you will have issues with your NiFi cluster will cluster coordinator and primary node elections.

If you found this answer addressed your questions, please accept as a solution.

Hope this helps,

Matt