Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

[#1]

Hello All, I've streaming data flowing into Kafka topics.

Can I add Broker & re-balance the topics while the data flows into Kafka topic ?

What are the best practices wrt. adding or removing Kafka Brokers while data is flowing into Kafka ?

Pls. advise.

Thanks!

[#2]

@mqureshi, @aromero,

@Neeraj Sabharwal, @Kuldeep Kulkarni, @Josh Elser - any ideas on this ?

[#3 · Accepted solution]

@Karan Alang

Yes, new brokers can be added while Kafka is online and partitions can be reassigned to these new brokers. One thing to remember is you are not creating new partitions. If you create new partitions, and assuming you have keyed messages where applications require data to be in order then you will lose the order guarantees because in a keyed message Kafka makes sure that a particular key always lands on a particular partition. When you add new partitions, you break this behavior.

But you can easily add new brokers and assign existing partitions to the new brokers in order to balance the cluster.

https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignParti...

[#4]

@mqureshi , thanks !