Community Articles

Find and share helpful community-sourced technical articles.
Cloudera Employee

Purpose of this article

Our purpose is to replace the nodes of Kafka Broker. We have two nodes newly added to the CDP PvC Base cluster. We will migrate two Kafka Brokers that were originally in use to these two new nodes. This article uses Cloudera CDP official documentation as a guide.

Summary

  • Environment

  • Steps of Broker Migration

    • Step-1: Startup the new broker as a member of the old cluster
    • Step-2: Stop both the new broker and the old broker that it is replacing
    • Step-3: Change broker.id of the new broker to the broker.id of the old one both in Cloudera Manager and in data directory/meta.properties.
    • Step-4: Startup the new broker.

Environment

Here are the currently in use Kafka Brokers:

kafka_brokers_original.png

Broker Host IP
1546345005 c3669-node3.coelab.cloudera.com 172.25.36.133
1546345009 c3669-node4.coelab.cloudera.com 172.25.36.207
1546345001 c3669-node2.coelab.cloudera.com 172.25.36.206

Steps of Broker Migration

  1. Startup the new broker as a member of the old cluster:kafka_add_broker-1.pngCM UI -> Kafka -> Instances -> Add Role Instanceskafka_add_broker-2.png

    We are only adding one broker at a time. We need to replace the brokers one by one.

    After adding host - host-10-17-102-138.coe.cloudera.com as Kafka Broker, it should not be starting yet. Now we will see there’s stale status in Kafka Broker cluster. So, deploy the client configurations by clicking the stale status icon on CM UI.

    Now, start the newly added broker on host-10-17-102-138.coe.cloudera.com.

  2. Stop both the new broker and the old broker that it is replacing.kafka_stop_new_and_old_broker-1.pngIn this example, I am replacing the old broker - 1546345005 on c3669-node3.coelab.cloudera.com with the new broker - 1546351866 on host-10-17-102-138.coe.cloudera.com.
  3. Change broker.id of the new broker to the broker.id of the old one both in Cloudera Manager and in data directory/meta.properties.kafka_change_brokerId-1.pngWe need to change the broker.id to the old one on the new Broker host.kafka_change_brokerId-2.png We also need to change the broker.id to the old one on CM UI. This will trigger a staleness computation. Therefore, deploy the client configuration as indicated by CM UI.
  4. Start up the new broker:kafka_start_new_broker-1.pngWe now can start the new Broker on host - host-10-17-102-138.coe.cloudera.com. Before starting, we would see WARNING message about: “lagging replicas” as of health check. After starting the new Broker, after a while (depends on the data volume on your cluster), the Kafka Broker cluster will become healthy again.

That’s all for Broker replacement/migration. Thanks for reading.

1,338 Views
Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.
Version history
Last update:
‎06-20-2021 07:57 PM
Updated by:
Contributors
Top Kudoed Authors