Support Questions

Find answers, ask questions, and share your expertise

Kafka Rolling Upgrade

avatar
Rising Star

Hi,

We are planning the rolling upgrade from HDP 2.4.0.0 to 2.5.3.0. No downtime during the upgrade is especially crucial for the Kafka cluster:

  1. Is the update of the Kafka brokers also rolling?
  2. Will clients (producer and cusumer) from Kafka 0.9.0.2.4 releases work with brokers from Kafka 0.10.0 releases?

If the answer for 1) and/or 2) is No - what is the best practice to guarantee no downtime?

Thank you in advance, Christian

1 ACCEPTED SOLUTION

avatar

@Christian Guegi

You can do the rolling upgrade without impacting downtime. Before starting the upgrade add these two properties to kafka config. You can add the below configs via ambari -> kafka -> Advanced custom properties.

1. inter.broker.protocol.version = 0.9.0 ( HDP 2.4 kafka version)

2. log.message.format.version = 0.9.0

once these properties are set you can upgrade one broker at a time. Once the broker upgraded and made it into ISR move onto next one to minimize any downtime clients might see.

Once the upgrade finished you can remove inter.broker.protocol.version .

Once you upgrade the clients to kafka_clients 0.10.0.1 version you can remove the log.message.format.version as well.

More details are on this page https://kafka.apache.org/0100/documentation.html#upgrade

View solution in original post

6 REPLIES 6

avatar
Super Collaborator

@Christian Guegi I believe there are constraints for Kafka based on versions and on whether a cluster has Kerberos enabled or not. I'll try to find someone to respond for 2.4 to 2.5.3.

avatar
Rising Star

Hi @lgeorge, fyi: the cluster is not kerberized.

avatar
Expert Contributor

@Christian Guegi You can go with manual upgrade of cluster and can upgrade kafka brokers one by one:

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_command-line-upgrade/content/upgrade-kafk...

avatar
Rising Star

Hi @lraheja, our HDP 2.4 cluster was installed with Ambari. Hence we must use Ambari Upgrade Guide to perform the HDP 2.4 to HDP 2.5.0 upgrade. I don't think a manual upgrade is an option.

avatar

@Christian Guegi

You can do the rolling upgrade without impacting downtime. Before starting the upgrade add these two properties to kafka config. You can add the below configs via ambari -> kafka -> Advanced custom properties.

1. inter.broker.protocol.version = 0.9.0 ( HDP 2.4 kafka version)

2. log.message.format.version = 0.9.0

once these properties are set you can upgrade one broker at a time. Once the broker upgraded and made it into ISR move onto next one to minimize any downtime clients might see.

Once the upgrade finished you can remove inter.broker.protocol.version .

Once you upgrade the clients to kafka_clients 0.10.0.1 version you can remove the log.message.format.version as well.

More details are on this page https://kafka.apache.org/0100/documentation.html#upgrade

avatar
Rising Star

FYI: With above description we were able to upgrade to version 2.5.3 without any Kafka cluster downtime. We only had some issues with a Kafka client written in Go.