We have a topic with 3 partitions where 1.5gb data getting pushed per minute but the whole data getting pushed to one partition(0) and other 2 partitions(1,2) not getting any data.
I believe Kafka by default take cares of distributing data between brokers and partitions.
Please help out with your thoughts.
Producers create new messages. In other publish/subscribe systems, these may be called publishers or writers. In general, a message will be produced to a specific topic. By default, the producer does not care what partition a specific message is written to and will balance messages over all partitions of a topic evenly. In some cases, the producer will direct messages to specific partitions. This is typically done using the message key and a partitioner that will generate a hash of the key and map it to a specific partition. This assures that all messages produced with a given key will get written to the same partition.
If you want to distribute the data among partitions make sure no key or partition is specified in your client.
Hope this helps.