Migrate brokers by modifying broker IDs in meta.properties

[#1]

Purpose of this article

Our purpose is to replace the nodes of Kafka Broker. We have two nodes newly added to the CDP PvC Base cluster. We will migrate two Kafka Brokers that were originally in use to these two new nodes. This article uses Cloudera CDP official documentation as a guide.

Summary

• Environment

• Steps of Broker Migration

  • Step-1: Startup the new broker as a member of the old cluster
  • Step-2: Stop both the new broker and the old broker that it is replacing
  • Step-3: Change broker.id of the new broker to the broker.id of the old one both in Cloudera Manager and in data directory/meta.properties.
  • Step-4: Startup the new broker.

Environment

Here are the currently in use Kafka Brokers:

Broker Host IP

1546345005	c3669-node3.coelab.cloudera.com	172.25.36.133
1546345009	c3669-node4.coelab.cloudera.com	172.25.36.207
1546345001	c3669-node2.coelab.cloudera.com	172.25.36.206

Steps of Broker Migration

1. Startup the new broker as a member of the old cluster:CM UI -> Kafka -> Instances -> Add Role Instances

   We are only adding one broker at a time. We need to replace the brokers one by one.

   After adding host - host-10-17-102-138.coe.cloudera.com as Kafka Broker, it should not be starting yet. Now we will see there’s stale status in Kafka Broker cluster. So, deploy the client configurations by clicking the stale status icon on CM UI.

   Now, start the newly added broker on host-10-17-102-138.coe.cloudera.com.

2. Stop both the new broker and the old broker that it is replacing.In this example, I am replacing the old broker - 1546345005 on c3669-node3.coelab.cloudera.com with the new broker - 1546351866 on host-10-17-102-138.coe.cloudera.com.
3. Change broker.id of the new broker to the broker.id of the old one both in Cloudera Manager and in data directory/meta.properties.We need to change the broker.id to the old one on the new Broker host. We also need to change the broker.id to the old one on CM UI. This will trigger a staleness computation. Therefore, deploy the client configuration as indicated by CM UI.
4. Start up the new broker:We now can start the new Broker on host - host-10-17-102-138.coe.cloudera.com. Before starting, we would see WARNING message about: “lagging replicas” as of health check. After starting the new Broker, after a while (depends on the data volume on your cluster), the Kafka Broker cluster will become healthy again.

That’s all for Broker replacement/migration. Thanks for reading.