Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Balancing kafka topics when new broker is added

avatar
Contributor

Hi,

We currently have 3 kafka brokers and about 250 topics. How do i re-balance the partitions and data when a new broker is added? Is running manual rebalance the only option? I was thinkin gof stopping a broker and then since we have 3 replications, it woul copy the topics in the stopped broker to new broker. Is that a good idea?

1 ACCEPTED SOLUTION

avatar

@Rajesh Reddy

Balancing data in kafka is not same as HDFS. You'd need to use partition re-assignment tool to move the partitions to new brokers.

Below is the detailed explanation from kafka documentation:

Adding servers to a Kafka cluster is easy, just assign them a unique broker id and start up Kafka on your new servers. However these new servers will not automatically be assigned any data partitions, so unless partitions are moved to them they won't be doing any work until new topics are created. So usually when you add machines to your cluster you will want to migrate some existing data to these machines.

The process of migrating data is manually initiated but fully automated. Under the covers what happens is that Kafka will add the new server as a follower of the partition it is migrating and allow it to fully replicate the existing data in that partition. When the new server has fully replicated the contents of this partition and joined the in-sync replica one of the existing replicas will delete their partition's data.

The partition reassignment tool can be used to move partitions across brokers. An ideal partition distribution would ensure even data load and partition sizes across all brokers. The partition reassignment tool does not have the capability to automatically study the data distribution in a Kafka cluster and move partitions around to attain an even load distribution. As such, the admin has to figure out which topics or partitions should be moved around.

View solution in original post

5 REPLIES 5

avatar

@Rajesh Reddy

Balancing data in kafka is not same as HDFS. You'd need to use partition re-assignment tool to move the partitions to new brokers.

Below is the detailed explanation from kafka documentation:

Adding servers to a Kafka cluster is easy, just assign them a unique broker id and start up Kafka on your new servers. However these new servers will not automatically be assigned any data partitions, so unless partitions are moved to them they won't be doing any work until new topics are created. So usually when you add machines to your cluster you will want to migrate some existing data to these machines.

The process of migrating data is manually initiated but fully automated. Under the covers what happens is that Kafka will add the new server as a follower of the partition it is migrating and allow it to fully replicate the existing data in that partition. When the new server has fully replicated the contents of this partition and joined the in-sync replica one of the existing replicas will delete their partition's data.

The partition reassignment tool can be used to move partitions across brokers. An ideal partition distribution would ensure even data load and partition sizes across all brokers. The partition reassignment tool does not have the capability to automatically study the data distribution in a Kafka cluster and move partitions around to attain an even load distribution. As such, the admin has to figure out which topics or partitions should be moved around.

avatar
Contributor

Hi @Sandeep Nemuri would this be the same when i add additional dara directories to kafka?

avatar

avatar
Contributor

Hi @Sandeep Nemuri That was helpful but what if i want to add additional disks rather than removing one and adding other??

avatar
Super Collaborator

I recommend to trigger the manual rebalancing via kafka-reassign-partition.sh, not by bringing down one node. In most scenarios, when you bring down one node and bring it back on afterwards, it is also not considered for balancing. And you might cause other issues as well.

https://community.hortonworks.com/content/supportkb/150063/how-to-add-new-kafka-brokers-to-a-cluster...