question Re: Kafka Rolling Upgrade in Support Questions

Kafka Rolling Upgrade

chri — Tue, 17 Jan 2017 20:22:01 GMT

Hi,

We are planning the rolling upgrade from HDP 2.4.0.0 to 2.5.3.0. No downtime during the upgrade is especially crucial for the Kafka cluster:

Is the update of the Kafka brokers also rolling?
Will clients (producer and cusumer) from Kafka 0.9.0.2.4 releases work with brokers from Kafka 0.10.0 releases?

If the answer for 1) and/or 2) is No - what is the best practice to guarantee no downtime?

Thank you in advance, Christian

Re: Kafka Rolling Upgrade

lgeorge — Wed, 18 Jan 2017 02:03:25 GMT

@Christian Guegi I believe there are constraints for Kafka based on versions and on whether a cluster has Kerberos enabled or not. I'll try to find someone to respond for 2.4 to 2.5.3.

Re: Kafka Rolling Upgrade

lraheja — Wed, 18 Jan 2017 07:51:14 GMT

@Christian Guegi You can go with manual upgrade of cluster and can upgrade kafka brokers one by one:

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_command-line-upgrade/content/upgrade-kafka-23.html

Re: Kafka Rolling Upgrade

chri — Wed, 18 Jan 2017 12:49:30 GMT

Hi @lgeorge, fyi: the cluster is not kerberized.

Re: Kafka Rolling Upgrade

chri — Wed, 18 Jan 2017 12:56:03 GMT

Hi @lraheja, our HDP 2.4 cluster was installed with Ambari. Hence we must use Ambari Upgrade Guide to perform the HDP 2.4 to HDP 2.5.0 upgrade. I don't think a manual upgrade is an option.

Re: Kafka Rolling Upgrade

schintalapani — Wed, 18 Jan 2017 13:43:43 GMT

@Christian Guegi

You can do the rolling upgrade without impacting downtime. Before starting the upgrade add these two properties to kafka config. You can add the below configs via ambari -> kafka -> Advanced custom properties.

1. inter.broker.protocol.version = 0.9.0 ( HDP 2.4 kafka version)

2. log.message.format.version = 0.9.0

once these properties are set you can upgrade one broker at a time. Once the broker upgraded and made it into ISR move onto next one to minimize any downtime clients might see.

Once the upgrade finished you can remove inter.broker.protocol.version .

Once you upgrade the clients to kafka_clients 0.10.0.1 version you can remove the log.message.format.version as well.

More details are on this page https://kafka.apache.org/0100/documentation.html#upgrade

Re: Kafka Rolling Upgrade

chri — Wed, 08 Mar 2017 22:27:03 GMT

FYI: With above description we were able to upgrade to version 2.5.3 without any Kafka cluster downtime. We only had some issues with a Kafka client written in Go.