Adding New Kafka Brokers: Question

[#1]

Hi all, this may be an easy question, but just wanted to verify.  When adding new brokers to the cluster, after installing the Kafka role/instance on the new hosts, should you start the new broker instances first before you restart the existing brokers?  Or does it matter?  This is CDP Private Cloud 7.1.7

Thanks

[#2]

AFAIU It should not matter. You can choose the rolling restart option as well. So CM can decide the sequence of the broker restarts. 

[#3 · Accepted solution]

@DataMike 

The broker restarting order will not cause any issues. But after adding new brokers you should rebalance topics/partitions between all available brokers for better performance.  Refer the following article for more details: 

https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/kafka-managing/topics/kafka-manage-cli-reassi...

If you found this response assisted with your query, please take a moment to log in and click on  KUDOS 🙂 & ”Accept as Solution" below this post.

Thank you.

[#4]

Thanks.  I think I can use Cruise Control to re-distribute the partitions across the new brokers.  Haven't tried yet though.

[#5]

 @DataMike 

Yes, you can use the CC APIs as per your requirements [1] [2]

[1] https://docs.cloudera.com/cdp-private-cloud-base/7.1.7/cctrl-managing/topics/cctrl-using-rest-api.ht...

[2] https://community.cloudera.com/t5/Customer/Frequently-Used-CRUISE-CONTROL-API-and-important-DOCs/ta-...

From the above articles you can use the following API to rebalance topics/partitions: 

curl -k --negotiate -u: -X POST "https://<CC FQDN>:8899/kafkacruisecontrol/rebalance?dryrun=false&rebalance_disk=true"

To avoid high CPU, memory and disk read write you can initiate rebalance process in batches. It will automatically create a batch and rebalance topics.

curl -X POST  "http://$HOSTNAME:8899/kafkacruisecontrol/rebalance?dryrun=true&concurrent_partition_movements_per_broker=10&concurrent_leader_movements=500"

If it will help you then please click on ”Accept as Solution" below this post.

Thank you.