Kafka Rolling Upgrade

[#1]

Hi,

We are planning the rolling upgrade from HDP 2.4.0.0 to 2.5.3.0. No downtime during the upgrade is especially crucial for the Kafka cluster:

1. Is the update of the Kafka brokers also rolling?
2. Will clients (producer and cusumer) from Kafka 0.9.0.2.4 releases work with brokers from Kafka 0.10.0 releases?

If the answer for 1) and/or 2) is No - what is the best practice to guarantee no downtime?

Thank you in advance, Christian

[#2]

@Christian Guegi I believe there are constraints for Kafka based on versions and on whether a cluster has Kerberos enabled or not. I'll try to find someone to respond for 2.4 to 2.5.3.

[#3]

Hi @lgeorge, fyi: the cluster is not kerberized.

[#4]

@Christian Guegi You can go with manual upgrade of cluster and can upgrade kafka brokers one by one:

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_command-line-upgrade/content/upgrade-kafk...

[#5]

Hi @lraheja, our HDP 2.4 cluster was installed with Ambari. Hence we must use Ambari Upgrade Guide to perform the HDP 2.4 to HDP 2.5.0 upgrade. I don't think a manual upgrade is an option.

[#6 · Accepted solution]

@Christian Guegi

You can do the rolling upgrade without impacting downtime. Before starting the upgrade add these two properties to kafka config. You can add the below configs via ambari -> kafka -> Advanced custom properties.

1. inter.broker.protocol.version = 0.9.0 ( HDP 2.4 kafka version)

2. log.message.format.version = 0.9.0

once these properties are set you can upgrade one broker at a time. Once the broker upgraded and made it into ISR move onto next one to minimize any downtime clients might see.

Once the upgrade finished you can remove inter.broker.protocol.version .

Once you upgrade the clients to kafka_clients 0.10.0.1 version you can remove the log.message.format.version as well.

More details are on this page https://kafka.apache.org/0100/documentation.html#upgrade

[#7]

FYI: With above description we were able to upgrade to version 2.5.3 without any Kafka cluster downtime. We only had some issues with a Kafka client written in Go.