Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

Solved Go to solution

Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

Expert Contributor
Hello All, I've streaming data flowing into Kafka topics.
Can I add Broker & re-balance the topics while the data flows into Kafka topic ?
What are the best practices wrt. adding or removing Kafka Brokers while data is flowing into Kafka ?
Pls. advise.
Thanks!
1 ACCEPTED SOLUTION

Accepted Solutions

Re: Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

Super Guru
@Karan Alang

Yes, new brokers can be added while Kafka is online and partitions can be reassigned to these new brokers. One thing to remember is you are not creating new partitions. If you create new partitions, and assuming you have keyed messages where applications require data to be in order then you will lose the order guarantees because in a keyed message Kafka makes sure that a particular key always lands on a particular partition. When you add new partitions, you break this behavior.

But you can easily add new brokers and assign existing partitions to the new brokers in order to balance the cluster.

https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignParti...

3 REPLIES 3
Highlighted

Re: Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

Expert Contributor

Re: Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

Super Guru
@Karan Alang

Yes, new brokers can be added while Kafka is online and partitions can be reassigned to these new brokers. One thing to remember is you are not creating new partitions. If you create new partitions, and assuming you have keyed messages where applications require data to be in order then you will lose the order guarantees because in a keyed message Kafka makes sure that a particular key always lands on a particular partition. When you add new partitions, you break this behavior.

But you can easily add new brokers and assign existing partitions to the new brokers in order to balance the cluster.

https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignParti...

Re: Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

Expert Contributor

@mqureshi , thanks !

Don't have an account?
Coming from Hortonworks? Activate your account here