Reply
New Contributor
Posts: 3
Registered: ‎08-05-2015

How KafkaSource can scale on Flume ?

Assuming I have a single Kafka topic with 4 partitions then I can scale by adding up to 4 KafkaSource, each one will read a partition of my single topic.

 

 

Now, assuming I have 4 topics, each one with 4 partitions. 

I have created a new KafkaSource able to read from multiple topics by using wildcard : https://github.com/guitoun3/flume/commit/85f2a8e4db422e3cd71c5ce14a48e576aad7c231

 

In this case, If I add new KafkaSource does it scale correctly ? I mean, do the available partition/topic will  be dispatched among my KafkaSource ?

If yes, in that case it means I can add up to 16 KafkaSource ?

Announcements
New solutions