Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Run multiple broker versions in the same cluster

Solved Go to solution

Run multiple broker versions in the same cluster

New Contributor

Hello,

We are currently running 4 kafka brokers v0.8 in our cluster (HDP-2.3.2-2950) and we would like to test some v0.10 brokers in the same cluster. Is this possible? I was thinking about installing manuallly in some machines of the cluster from HDP-2.6.0 repo and apart from Ambari, but I don't know how this will affect zookeepers and if the whole of brokers will be seen as 2 groups of brokers and will work indepently.

I would need advice :-)

Regards,

Silvio

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Run multiple broker versions in the same cluster

@Silvio del Val

Yes, you can install and manage the second set of Kafka brokers manually (not through Ambari). As for Zookeeper, it can manage multiple Kafka clusters. Just "chroot" them differently

{zookeeperHost}:{portnumber}/{kafkacluster1}

&

{zookeeperHost}:{portnumber}/{kafkacluster2}

Take a look at the link below on how zookeeper manages sessions and on chroot.

https://zookeeper.apache.org/doc/r3.2.2/zookeeperProgrammers.html#ch_zkSessions

To do this:

1) Create the chroot in Zookeeper with the below commands

zkCli.sh -server yourZookeeperHost:2181
# note the empty brackets below are _required_
create /kafkaCluster1 []

This will create a path called "kafkaCluster1". You can verify this by running the "ls/" command.

(* replace "yourZookeeperHost" and the port number with the appropriate values from you environment)

2) For the Kafka brokers configs, under "zookeeper.connect", add a "chroot" path which will make all kafka data for this cluster appear under that particular path. To do this give a connection string in the form kafka_v08_1:port1,kafka_v08_2:port2,kafka_v08_3:port3 yourZookeeperHost:2181/kafkaCluster1 which would put all this cluster's data under the path yourZookeeperHost:2181/kafkaCluster1.

3) For the Kafka consumer configs, under "zookeeper.connect", input the same as you did in step 2 for the brokers

4) Repeat steps 1-3 for your second Kafka cluster using a different "chroot" path

For more explanation of Kafka configs take a look at the below link.

https://kafka.apache.org/08/documentation/#configuration

9 REPLIES 9

Re: Run multiple broker versions in the same cluster

@Silvio del Val

Yes, you can install and manage the second set of Kafka brokers manually (not through Ambari). As for Zookeeper, it can manage multiple Kafka clusters. Just "chroot" them differently

{zookeeperHost}:{portnumber}/{kafkacluster1}

&

{zookeeperHost}:{portnumber}/{kafkacluster2}

Take a look at the link below on how zookeeper manages sessions and on chroot.

https://zookeeper.apache.org/doc/r3.2.2/zookeeperProgrammers.html#ch_zkSessions

To do this:

1) Create the chroot in Zookeeper with the below commands

zkCli.sh -server yourZookeeperHost:2181
# note the empty brackets below are _required_
create /kafkaCluster1 []

This will create a path called "kafkaCluster1". You can verify this by running the "ls/" command.

(* replace "yourZookeeperHost" and the port number with the appropriate values from you environment)

2) For the Kafka brokers configs, under "zookeeper.connect", add a "chroot" path which will make all kafka data for this cluster appear under that particular path. To do this give a connection string in the form kafka_v08_1:port1,kafka_v08_2:port2,kafka_v08_3:port3 yourZookeeperHost:2181/kafkaCluster1 which would put all this cluster's data under the path yourZookeeperHost:2181/kafkaCluster1.

3) For the Kafka consumer configs, under "zookeeper.connect", input the same as you did in step 2 for the brokers

4) Repeat steps 1-3 for your second Kafka cluster using a different "chroot" path

For more explanation of Kafka configs take a look at the below link.

https://kafka.apache.org/08/documentation/#configuration

Re: Run multiple broker versions in the same cluster

New Contributor

Well, we have currently running 4 v0.8 brokers with default config. How should I config Zookeepers to specify new brokers in second cluster?

Re: Run multiple broker versions in the same cluster

New Contributor

I enter ZkCli but I don't know how to do

Re: Run multiple broker versions in the same cluster

I've updated my response to include the steps.

Re: Run multiple broker versions in the same cluster

New Contributor

So, do I have to chroot existing cluster too? This 4 brokers group is in production and I can´t stop them.

Apart from this, I'll install 3 new brokers in a second cluster but this will be v0.10. Can I use same zookeepers? I have HDP-2.3.2-2950 and I'll install this second group from HDP-2.6 repositories

Re: Run multiple broker versions in the same cluster

You do not need to change the path for your existing cluster. Leave it as is. Chroot the second/new cluster. As for versions, you should be ok. To make your config life easier, I suggest you not install both Kafka versions on the same nodes though.

Re: Run multiple broker versions in the same cluster

New Contributor

Great!! Well, I was thinkg about installing these v0.10 brokers in other empty machines. I suppose it's imposible to manage these v0.10 brokers in same Ambari, right? Maybe I should have both HDP versions installed, but only one of them is current...well, to difficult maybe.

Anyway, thank you very very much for your answers, you saved my life :-)

I'll mark your answer as accepted :-)

Re: Run multiple broker versions in the same cluster

Glad I could help. No, unfortunately you won't be able to manage both clusters with the same Ambari.

Re: Run multiple broker versions in the same cluster

New Contributor

Well, we have currently running 4 v0.8 brokers with default config. How should I config Zookeepers to specify new brokers in second cluster?