Support Questions
Find answers, ask questions, and share your expertise

Kafka Rolling Upgrade

Contributor

Hi,

We are planning the rolling upgrade from HDP 2.4.0.0 to 2.5.3.0. No downtime during the upgrade is especially crucial for the Kafka cluster:

  1. Is the update of the Kafka brokers also rolling?
  2. Will clients (producer and cusumer) from Kafka 0.9.0.2.4 releases work with brokers from Kafka 0.10.0 releases?

If the answer for 1) and/or 2) is No - what is the best practice to guarantee no downtime?

Thank you in advance, Christian

1 ACCEPTED SOLUTION

Accepted Solutions

@Christian Guegi

You can do the rolling upgrade without impacting downtime. Before starting the upgrade add these two properties to kafka config. You can add the below configs via ambari -> kafka -> Advanced custom properties.

1. inter.broker.protocol.version = 0.9.0 ( HDP 2.4 kafka version)

2. log.message.format.version = 0.9.0

once these properties are set you can upgrade one broker at a time. Once the broker upgraded and made it into ISR move onto next one to minimize any downtime clients might see.

Once the upgrade finished you can remove inter.broker.protocol.version .

Once you upgrade the clients to kafka_clients 0.10.0.1 version you can remove the log.message.format.version as well.

More details are on this page https://kafka.apache.org/0100/documentation.html#upgrade

View solution in original post

6 REPLIES 6

Expert Contributor

@Christian Guegi I believe there are constraints for Kafka based on versions and on whether a cluster has Kerberos enabled or not. I'll try to find someone to respond for 2.4 to 2.5.3.

Contributor

Hi @lgeorge, fyi: the cluster is not kerberized.

Contributor

@Christian Guegi You can go with manual upgrade of cluster and can upgrade kafka brokers one by one:

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_command-line-upgrade/content/upgrade-kafk...

Contributor

Hi @lraheja, our HDP 2.4 cluster was installed with Ambari. Hence we must use Ambari Upgrade Guide to perform the HDP 2.4 to HDP 2.5.0 upgrade. I don't think a manual upgrade is an option.

@Christian Guegi

You can do the rolling upgrade without impacting downtime. Before starting the upgrade add these two properties to kafka config. You can add the below configs via ambari -> kafka -> Advanced custom properties.

1. inter.broker.protocol.version = 0.9.0 ( HDP 2.4 kafka version)

2. log.message.format.version = 0.9.0

once these properties are set you can upgrade one broker at a time. Once the broker upgraded and made it into ISR move onto next one to minimize any downtime clients might see.

Once the upgrade finished you can remove inter.broker.protocol.version .

Once you upgrade the clients to kafka_clients 0.10.0.1 version you can remove the log.message.format.version as well.

More details are on this page https://kafka.apache.org/0100/documentation.html#upgrade

View solution in original post

Contributor

FYI: With above description we were able to upgrade to version 2.5.3 without any Kafka cluster downtime. We only had some issues with a Kafka client written in Go.