08-05-2015 02:32 AM
Assuming I have a single Kafka topic with 4 partitions then I can scale by adding up to 4 KafkaSource, each one will read a partition of my single topic.
Now, assuming I have 4 topics, each one with 4 partitions.
I have created a new KafkaSource able to read from multiple topics by using wildcard : https://github.com/guitoun3/flume/commit/85f2a8e4db422e3cd71c5ce14a48e576aad7c231
In this case, If I add new KafkaSource does it scale correctly ? I mean, do the available partition/topic will be dispatched among my KafkaSource ?
If yes, in that case it means I can add up to 16 KafkaSource ?