Support Questions

Find answers, ask questions, and share your expertise

Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

avatar
Expert Contributor
Hello All, I've streaming data flowing into Kafka topics.
Can I add Broker & re-balance the topics while the data flows into Kafka topic ?
What are the best practices wrt. adding or removing Kafka Brokers while data is flowing into Kafka ?
Pls. advise.
Thanks!
1 ACCEPTED SOLUTION

avatar
Super Guru
@Karan Alang

Yes, new brokers can be added while Kafka is online and partitions can be reassigned to these new brokers. One thing to remember is you are not creating new partitions. If you create new partitions, and assuming you have keyed messages where applications require data to be in order then you will lose the order guarantees because in a keyed message Kafka makes sure that a particular key always lands on a particular partition. When you add new partitions, you break this behavior.

But you can easily add new brokers and assign existing partitions to the new brokers in order to balance the cluster.

https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignParti...

View solution in original post

3 REPLIES 3

avatar
Expert Contributor

avatar
Super Guru
@Karan Alang

Yes, new brokers can be added while Kafka is online and partitions can be reassigned to these new brokers. One thing to remember is you are not creating new partitions. If you create new partitions, and assuming you have keyed messages where applications require data to be in order then you will lose the order guarantees because in a keyed message Kafka makes sure that a particular key always lands on a particular partition. When you add new partitions, you break this behavior.

But you can easily add new brokers and assign existing partitions to the new brokers in order to balance the cluster.

https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignParti...

avatar
Expert Contributor

@mqureshi , thanks !