How to choose number of kafka partitions

[#1]

Right now we have only one topic , with 3 partitions and with a replicaiton factor of 3. My log.dirs has 4 mount point or paths or directories.

Only 1 mount point is being used out of 4 . Other 3 mount point or paths or directories , mentioned in log.dirs are not been used.

How can i make all the mount / directories , get used.

Thanks

JJ

[#2]

@Jacqualin jasmin

Can you locate the kafka server.properties ? The log dir parameter which is a comma separeted should list of directories is where kafka stores the log files. see example below of 3 different log dirs

log.dirs=/xxx/xxx/xxx,/bbb/bbb/bbb,/fff/fff/fff

Ensure that the directories you configure are listed and restart kafka.

[#3]

Yes my log.dirs has 4 comma separated directories. Out of 4, only one directory is getting used. How can we make sure all the 4 directories are equally be used.

Right now I have 3 brokers and my topic has 3 partitions. 3 partitions are spread across 3 brokers.

my question is why all the directories were not getting filled ( why only one ). Does the number of partitions of a topic should depend on the number of directories in log.dirs?

Regards

JJ

[#4]

@Jacqualin jasmin - what is your partitioning scheme? Keep in mind:

The default partitioning strategy has following flow while deciding partition,

• If a partition is specified in the record, use it
• If no partition is specified but a key is present choose a partition based on a hash of the key
• If no partition or key is present choose a partition in a round-robin fashion

(that is from this HCC article)

Is it possible you're pushing records through one partition?