Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Kafka topic data not partitioned data among brokers

Highlighted

Kafka topic data not partitioned data among brokers

New Contributor

Hello,

We have a topic with 3 partitions where 1.5gb data getting pushed per minute but the whole data getting pushed to one partition(0) and other 2 partitions(1,2) not getting any data.

I believe Kafka by default take cares of distributing data between brokers and partitions.

Please help out with your thoughts.


Thanks,

Arshad

1 REPLY 1

Re: Kafka topic data not partitioned data among brokers

Contributor

Hello,

 

Producers create new messages. In other publish/subscribe systems, these may be called publishers or writers. In general, a message will be produced to a specific topic. By default, the producer does not care what partition a specific message is written to and will balance messages over all partitions of a topic evenly. In some cases, the producer will direct messages to specific partitions. This is typically done using the message key and a partitioner that will generate a hash of the key and map it to a specific partition. This assures that all messages produced with a given key will get written to the same partition.

 

If you want to distribute the data among partitions make sure no key or partition is specified in your client.

 

Hope this helps.

Regards,

Manuel.