configuration of multiple zookeeper quorums on a single HDP 2.4.2 Kerberized Cluster - Need Help pls

[#1]

Hi I am planning to build a HDP 2.4.2 Kerberized cluster via Ambari Blue-Prints and I am going to change the blue print to have 6 Zookeepers. The reason why I am having 6 zk's is I want to have two ZK quorums with 3 ZK's, 1 quorum I want to use for HDFS NN HA, Hbase and other services except for Kafka and for Kafka alone I want to have other ZK Quorum dedicated. I am assuming when I build the cluster with 6 ZK's initially I guess it will create only 1 ZK quorum with 6 ZK's in it. Can I change it to have to 2 ZK quorums after cluster installation from zkcli? or is there an option in Ambari blue print itself to create 2 ZK quorums with 3 ZK servers in each quorum? Please advice! Thanks

[#2 · Accepted solution]

You are right, because Ambari supports only one ZK quorum, if you have 6 ZKs in your blueprint you will end up with a single 6-node ZK quorum, and you cannot change that using zkcli. Instead you can try:

• 1 cluster (a): Install the cluster from your blueprint without Kafka and with only 3 ZK nodes. When the cluster is up and running install the second, Kafka ZK quorum manually, you can find instructions here. Finally, add Kafka using Ambari and set its ZK quorum to the one you istalled manually.
• 1 cluster (b): Install the cluster from blueprint including Kafka and 3 ZK nodes. Then install another ZK manually using the above link, and change Kafka settings to use the new Zookeeper. I'd avoid this solution because the first Kafka will be "poluted" by Kafka ZK directories.
• 1 cluster (c), your solution with 6 ZK nodes: Remove 3 ZK nodes using Ambari and then install another ZK manually and change Kafka settings like in 1(b).
• 2 clusters: Install one cluster with all services but without Kafka and its Zookeeper, and one more Kafka-only cluster having only Kafka and its Zookeeper. This is the easiest solution because you can automate cluster deployment using Ambari, and you can monitor all your components using Ambari, but you will have 2 clusters.

[#3]

@Predrag Minovic

Great options. It looks like from all the options above the 2nd ZK quorum should be installed manually outside Ambari and configure the Kafka accordingly? If that's the case when I do upgrades in future on this cluster I have to take care of 2nd manual ZK quorum upgrade as a separate effort rt? And I like the 2 clusters solution but what if some business logic on cluster 1 is dependent on kafka on cluster 2? In that case I guess "2 clusters " solution will not work rt? Please confirm! Thanks Sri.

[#4]

@Pred, I'm Vaidya from SCB. For this same scenario above we have 6 Zookeeper in an ensemble. All 6 zookeepers are handled through Ambari where I have configured 3 for NN HA and other components whereas 3 ZK dedicatedly for Kafka. The environment is running fine. But how does this is handled by the cluster where I can have only 1 Zookeeper leader in a quorum ? Please clarify.

[#5]

Hi Vaidya, if in Ambari Zookeeper summary page you see 6 ZK servers, then all 6 are running in the same ensemble and have one leader and 5 followers. You can check that using your favorite tool to run the same shell command on all 6 master nodes, and use "echo stat | nc localhost 2181 | grep Mode" as the command.