question Kafka Partitioning Class : Clarification in Archives of Support Questions (Read Only)

Kafka Partitioning Class : Clarification

Jagatheeshr — Fri, 16 Oct 2015 17:20:51 GMT

We create Kafka Partition using kafka-topics.sh. Would like to validate my understanding is correct. Partitioning strategy is actually performed by Kafka rather than Producer. As per http://kafka.apache.org/07/configuration.html (Important configuration properties for Kafka broker:) producer.DefaultPartitioner<T> - uses the partitioning strategy hash(key)%num_partitions. If key is null, then it picks a random partition. Producer would either pass a key or not which is then used by Kafka Broker to determine partition strategy whether to use Hash Partitioning or Random Key.

Re: Kafka Partitioning Class : Clarification

schintalapani — Sun, 18 Oct 2015 00:10:35 GMT

This is old config file http://kafka.apache.org/07/configuration.html version 0.7

for the new configuration you should look at https://kafka.apache.org/08/configuration.html

By using kafka-topics.sh you create how many partitions you would like to have in a topic. Usually this should be determined by how much parallelism you would like to have on the consumer side to read from the topic.

Kafka doesn't determine how you distribute the data into the topic partitions. That depends on the producer. As you said if key is null it does a round-robin not random. If the key is provided it does a hash based distribution. All of this happens on the producer side not in kafka broker.

Re: Kafka Partitioning Class : Clarification

Jagatheeshr — Sun, 18 Oct 2015 04:51:09 GMT

Thanks @schintalapani@hortonworks.com. So i see this as a change in the way kafka works from .7 to .8 .