question Re: Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics in Archives of Support Questions (Read Only)

Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

karan_alang1 — Wed, 14 Jun 2017 05:00:51 GMT

Hello All, I've streaming data flowing into Kafka topics.

Can I add Broker & re-balance the topics while the data flows into Kafka topic ?

What are the best practices wrt. adding or removing Kafka Brokers while data is flowing into Kafka ?

Pls. advise.

Thanks!

Re: Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

karan_alang1 — Thu, 15 Jun 2017 00:24:18 GMT

@mqureshi, @aromero,

@Neeraj Sabharwal, @Kuldeep Kulkarni, @Josh Elser - any ideas on this ?

Re: Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

mqureshi — Thu, 15 Jun 2017 00:54:49 GMT

@Karan Alang

Yes, new brokers can be added while Kafka is online and partitions can be reassigned to these new brokers. One thing to remember is you are not creating new partitions. If you create new partitions, and assuming you have keyed messages where applications require data to be in order then you will lose the order guarantees because in a keyed message Kafka makes sure that a particular key always lands on a particular partition. When you add new partitions, you break this behavior.

But you can easily add new brokers and assign existing partitions to the new brokers in order to balance the cluster.

https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool

Re: Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

karan_alang1 — Fri, 16 Jun 2017 01:57:38 GMT

@mqureshi , thanks !