question Re: Run multiple broker versions in the same cluster in Archives of Support Questions (Read Only)

Run multiple broker versions in the same cluster

silvio_delval — Wed, 10 May 2017 20:24:20 GMT

Hello,

We are currently running 4 kafka brokers v0.8 in our cluster (HDP-2.3.2-2950) and we would like to test some v0.10 brokers in the same cluster. Is this possible? I was thinking about installing manuallly in some machines of the cluster from HDP-2.6.0 repo and apart from Ambari, but I don't know how this will affect zookeepers and if the whole of brokers will be seen as 2 groups of brokers and will work indepently.

I would need advice 🙂

Regards,

Silvio

Re: Run multiple broker versions in the same cluster

egarelnabi — Wed, 10 May 2017 22:34:36 GMT

@Silvio del Val

Yes, you can install and manage the second set of Kafka brokers manually (not through Ambari). As for Zookeeper, it can manage multiple Kafka clusters. Just "chroot" them differently

{zookeeperHost}:{portnumber}/{kafkacluster1}

{zookeeperHost}:{portnumber}/{kafkacluster2}

Take a look at the link below on how zookeeper manages sessions and on chroot.

https://zookeeper.apache.org/doc/r3.2.2/zookeeperProgrammers.html#ch_zkSessions

To do this:

1) Create the chroot in Zookeeper with the below commands

zkCli.sh -server yourZookeeperHost:2181

# note the empty brackets below are _required_
create /kafkaCluster1 []

This will create a path called "kafkaCluster1". You can verify this by running the "ls/" command.

(* replace "yourZookeeperHost" and the port number with the appropriate values from you environment)

2) For the Kafka brokers configs, under "zookeeper.connect", add a "chroot" path which will make all kafka data for this cluster appear under that particular path. To do this give a connection string in the form kafka_v08_1:port1,kafka_v08_2:port2,kafka_v08_3:port3 yourZookeeperHost:2181/kafkaCluster1 which would put all this cluster's data under the path yourZookeeperHost:2181/kafkaCluster1.

3) For the Kafka consumer configs, under "zookeeper.connect", input the same as you did in step 2 for the brokers

4) Repeat steps 1-3 for your second Kafka cluster using a different "chroot" path

For more explanation of Kafka configs take a look at the below link.

https://kafka.apache.org/08/documentation/#configuration

Re: Run multiple broker versions in the same cluster

silvio_delval — Wed, 10 May 2017 22:49:30 GMT

Well, we have currently running 4 v0.8 brokers with default config. How should I config Zookeepers to specify new brokers in second cluster?

Re: Run multiple broker versions in the same cluster

silvio_delval — Wed, 10 May 2017 23:04:38 GMT

Well, we have currently running 4 v0.8 brokers with default config. How should I config Zookeepers to specify new brokers in second cluster?

Re: Run multiple broker versions in the same cluster

silvio_delval — Wed, 10 May 2017 23:05:15 GMT

I enter ZkCli but I don't know how to do

Re: Run multiple broker versions in the same cluster

egarelnabi — Wed, 10 May 2017 23:18:57 GMT

I've updated my response to include the steps.

Re: Run multiple broker versions in the same cluster

silvio_delval — Thu, 11 May 2017 17:12:05 GMT

So, do I have to chroot existing cluster too? This 4 brokers group is in production and I can´t stop them.

Apart from this, I'll install 3 new brokers in a second cluster but this will be v0.10. Can I use same zookeepers? I have HDP-2.3.2-2950 and I'll install this second group from HDP-2.6 repositories

Re: Run multiple broker versions in the same cluster

egarelnabi — Thu, 11 May 2017 22:03:03 GMT

You do not need to change the path for your existing cluster. Leave it as is. Chroot the second/new cluster. As for versions, you should be ok. To make your config life easier, I suggest you not install both Kafka versions on the same nodes though.

Re: Run multiple broker versions in the same cluster

silvio_delval — Thu, 11 May 2017 22:52:23 GMT

Great!! Well, I was thinkg about installing these v0.10 brokers in other empty machines. I suppose it's imposible to manage these v0.10 brokers in same Ambari, right? Maybe I should have both HDP versions installed, but only one of them is current...well, to difficult maybe.

Anyway, thank you very very much for your answers, you saved my life 🙂

I'll mark your answer as accepted 🙂

Re: Run multiple broker versions in the same cluster

egarelnabi — Thu, 11 May 2017 22:55:15 GMT

Glad I could help. No, unfortunately you won't be able to manage both clusters with the same Ambari.