Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Run multiple broker versions in the same cluster

avatar
Rising Star

Hello,

We are currently running 4 kafka brokers v0.8 in our cluster (HDP-2.3.2-2950) and we would like to test some v0.10 brokers in the same cluster. Is this possible? I was thinking about installing manuallly in some machines of the cluster from HDP-2.6.0 repo and apart from Ambari, but I don't know how this will affect zookeepers and if the whole of brokers will be seen as 2 groups of brokers and will work indepently.

I would need advice 🙂

Regards,

Silvio

1 ACCEPTED SOLUTION

avatar

@Silvio del Val

Yes, you can install and manage the second set of Kafka brokers manually (not through Ambari). As for Zookeeper, it can manage multiple Kafka clusters. Just "chroot" them differently

{zookeeperHost}:{portnumber}/{kafkacluster1}

&

{zookeeperHost}:{portnumber}/{kafkacluster2}

Take a look at the link below on how zookeeper manages sessions and on chroot.

https://zookeeper.apache.org/doc/r3.2.2/zookeeperProgrammers.html#ch_zkSessions

To do this:

1) Create the chroot in Zookeeper with the below commands

zkCli.sh -server yourZookeeperHost:2181
# note the empty brackets below are _required_
create /kafkaCluster1 []

This will create a path called "kafkaCluster1". You can verify this by running the "ls/" command.

(* replace "yourZookeeperHost" and the port number with the appropriate values from you environment)

2) For the Kafka brokers configs, under "zookeeper.connect", add a "chroot" path which will make all kafka data for this cluster appear under that particular path. To do this give a connection string in the form kafka_v08_1:port1,kafka_v08_2:port2,kafka_v08_3:port3 yourZookeeperHost:2181/kafkaCluster1 which would put all this cluster's data under the path yourZookeeperHost:2181/kafkaCluster1.

3) For the Kafka consumer configs, under "zookeeper.connect", input the same as you did in step 2 for the brokers

4) Repeat steps 1-3 for your second Kafka cluster using a different "chroot" path

For more explanation of Kafka configs take a look at the below link.

https://kafka.apache.org/08/documentation/#configuration

View solution in original post

9 REPLIES 9

avatar

@Silvio del Val

Yes, you can install and manage the second set of Kafka brokers manually (not through Ambari). As for Zookeeper, it can manage multiple Kafka clusters. Just "chroot" them differently

{zookeeperHost}:{portnumber}/{kafkacluster1}

&

{zookeeperHost}:{portnumber}/{kafkacluster2}

Take a look at the link below on how zookeeper manages sessions and on chroot.

https://zookeeper.apache.org/doc/r3.2.2/zookeeperProgrammers.html#ch_zkSessions

To do this:

1) Create the chroot in Zookeeper with the below commands

zkCli.sh -server yourZookeeperHost:2181
# note the empty brackets below are _required_
create /kafkaCluster1 []

This will create a path called "kafkaCluster1". You can verify this by running the "ls/" command.

(* replace "yourZookeeperHost" and the port number with the appropriate values from you environment)

2) For the Kafka brokers configs, under "zookeeper.connect", add a "chroot" path which will make all kafka data for this cluster appear under that particular path. To do this give a connection string in the form kafka_v08_1:port1,kafka_v08_2:port2,kafka_v08_3:port3 yourZookeeperHost:2181/kafkaCluster1 which would put all this cluster's data under the path yourZookeeperHost:2181/kafkaCluster1.

3) For the Kafka consumer configs, under "zookeeper.connect", input the same as you did in step 2 for the brokers

4) Repeat steps 1-3 for your second Kafka cluster using a different "chroot" path

For more explanation of Kafka configs take a look at the below link.

https://kafka.apache.org/08/documentation/#configuration

avatar
Rising Star

Well, we have currently running 4 v0.8 brokers with default config. How should I config Zookeepers to specify new brokers in second cluster?

avatar
Rising Star

I enter ZkCli but I don't know how to do

avatar

I've updated my response to include the steps.

avatar
Rising Star

So, do I have to chroot existing cluster too? This 4 brokers group is in production and I can´t stop them.

Apart from this, I'll install 3 new brokers in a second cluster but this will be v0.10. Can I use same zookeepers? I have HDP-2.3.2-2950 and I'll install this second group from HDP-2.6 repositories

avatar

You do not need to change the path for your existing cluster. Leave it as is. Chroot the second/new cluster. As for versions, you should be ok. To make your config life easier, I suggest you not install both Kafka versions on the same nodes though.

avatar
Rising Star

Great!! Well, I was thinkg about installing these v0.10 brokers in other empty machines. I suppose it's imposible to manage these v0.10 brokers in same Ambari, right? Maybe I should have both HDP versions installed, but only one of them is current...well, to difficult maybe.

Anyway, thank you very very much for your answers, you saved my life 🙂

I'll mark your answer as accepted 🙂

avatar

Glad I could help. No, unfortunately you won't be able to manage both clusters with the same Ambari.

avatar
Rising Star

Well, we have currently running 4 v0.8 brokers with default config. How should I config Zookeepers to specify new brokers in second cluster?