Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Kafka Rolling Upgrade

Solved Go to solution

Kafka Rolling Upgrade

Contributor

Hi,

We are planning the rolling upgrade from HDP 2.4.0.0 to 2.5.3.0. No downtime during the upgrade is especially crucial for the Kafka cluster:

  1. Is the update of the Kafka brokers also rolling?
  2. Will clients (producer and cusumer) from Kafka 0.9.0.2.4 releases work with brokers from Kafka 0.10.0 releases?

If the answer for 1) and/or 2) is No - what is the best practice to guarantee no downtime?

Thank you in advance, Christian

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Kafka Rolling Upgrade

@Christian Guegi

You can do the rolling upgrade without impacting downtime. Before starting the upgrade add these two properties to kafka config. You can add the below configs via ambari -> kafka -> Advanced custom properties.

1. inter.broker.protocol.version = 0.9.0 ( HDP 2.4 kafka version)

2. log.message.format.version = 0.9.0

once these properties are set you can upgrade one broker at a time. Once the broker upgraded and made it into ISR move onto next one to minimize any downtime clients might see.

Once the upgrade finished you can remove inter.broker.protocol.version .

Once you upgrade the clients to kafka_clients 0.10.0.1 version you can remove the log.message.format.version as well.

More details are on this page https://kafka.apache.org/0100/documentation.html#upgrade

View solution in original post

6 REPLIES 6
Highlighted

Re: Kafka Rolling Upgrade

Expert Contributor

@Christian Guegi I believe there are constraints for Kafka based on versions and on whether a cluster has Kerberos enabled or not. I'll try to find someone to respond for 2.4 to 2.5.3.

Highlighted

Re: Kafka Rolling Upgrade

Contributor

Hi @lgeorge, fyi: the cluster is not kerberized.

Highlighted

Re: Kafka Rolling Upgrade

Contributor

@Christian Guegi You can go with manual upgrade of cluster and can upgrade kafka brokers one by one:

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_command-line-upgrade/content/upgrade-kafk...

Highlighted

Re: Kafka Rolling Upgrade

Contributor

Hi @lraheja, our HDP 2.4 cluster was installed with Ambari. Hence we must use Ambari Upgrade Guide to perform the HDP 2.4 to HDP 2.5.0 upgrade. I don't think a manual upgrade is an option.

Highlighted

Re: Kafka Rolling Upgrade

@Christian Guegi

You can do the rolling upgrade without impacting downtime. Before starting the upgrade add these two properties to kafka config. You can add the below configs via ambari -> kafka -> Advanced custom properties.

1. inter.broker.protocol.version = 0.9.0 ( HDP 2.4 kafka version)

2. log.message.format.version = 0.9.0

once these properties are set you can upgrade one broker at a time. Once the broker upgraded and made it into ISR move onto next one to minimize any downtime clients might see.

Once the upgrade finished you can remove inter.broker.protocol.version .

Once you upgrade the clients to kafka_clients 0.10.0.1 version you can remove the log.message.format.version as well.

More details are on this page https://kafka.apache.org/0100/documentation.html#upgrade

View solution in original post

Highlighted

Re: Kafka Rolling Upgrade

Contributor

FYI: With above description we were able to upgrade to version 2.5.3 without any Kafka cluster downtime. We only had some issues with a Kafka client written in Go.

Don't have an account?
Coming from Hortonworks? Activate your account here