question Re: Is it possible to have two versions of Kafka in the same CDH cluster ? in Support Questions

Is it possible to have two versions of Kafka in the same CDH cluster ?

iamabug — Tue, 20 Aug 2019 04:47:27 GMT

We already have a Kafka in our CDH cluster and we wish to set up a newer version of Kafka in the same cluster. I have tried adding new Kafka parcel in CM and it seems only one version of Kafka parcel can be enabled at the same time. I wonder whether it is possible. Thanks.

Re: Is it possible to have two versions of Kafka in the same CDH cluster ?

w@leed — Tue, 20 Aug 2019 21:06:44 GMT

Hi @iamabug

This is not possible unfortunately. You could only have one parcel for a specific service activated at any given time. You would need to have a separate cluster managed by a different Cloudera Manager installation in order to activate a different Kafka parcel version.

Re: Is it possible to have two versions of Kafka in the same CDH cluster ?

iamabug — Wed, 21 Aug 2019 01:04:55 GMT

Thanks. Do you mean two clusters managed by one CM or two clusters managed by two different CMs ? I believe it is the former.

Re: Is it possible to have two versions of Kafka in the same CDH cluster ?

w@leed — Wed, 21 Aug 2019 03:04:50 GMT

Two different clusters managed by two different Cloudera Managers.

Re: Is it possible to have two versions of Kafka in the same CDH cluster ?

iamabug — Wed, 21 Aug 2019 07:25:29 GMT

I managed to install a second cluster, which only includes Kafak and Zookeeper, as shown in the figure below. w@leed

Re: Is it possible to have two versions of Kafka in the same CDH cluster ?

w@leed — Wed, 21 Aug 2019 13:04:33 GMT

Hi @iamabug

Yes, you can definitely have two different clusters managed by the same Cloudera Manager instance. In return, you can have different CDH versions for the two clusters, however, my initial thoughts was that you wouldn't be able to activate two different Kafka parcel versions, for example CDK 3.0 on Cluster A and CDK 2.2 on Cluster B.

But I need to double check this because I have some doubts if that is in fact the case in CDH 5.x or not. In CDH 6.x you're bound to a specific version of Kafka depending on your CDH version since that's not released as a separate customer service so my response was geared more towards C6 rather than C5.

Do you have two different versions of Kafka running on cluster 1 and cluster 2?

Re: Is it possible to have two versions of Kafka in the same CDH cluster ?

iamabug — Wed, 21 Aug 2019 13:04:44 GMT

I have CDK 3.1.0 running on cluster 1 and CDK 4.1.0 running on cluster 2. And CDH version is 5.15.1.

Re: Is it possible to have two versions of Kafka in the same CDH cluster ?

w@leed — Wed, 21 Aug 2019 13:06:55 GMT

Thanks for confirming.

Yes, I see that this is in fact possible in CDH 5 as well when I did a quick check.

Just remember that starting with CDH 6.0, both Spark 2.x and Kafka are bundled in CDH, so the only way you would have two different versions is by running two different CDH versions from Cloudera Manager.

Re: Is it possible to have two versions of Kafka in the same CDH cluster ?

iamabug — Wed, 21 Aug 2019 13:10:24 GMT

Now I am really clear about the situation. Thanks a lot for your replies.