Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to choose number of kafka partitions

How to choose number of kafka partitions

New Contributor

Right now we have only one topic , with 3 partitions and with a replicaiton factor of 3. My log.dirs has 4 mount point or paths or directories.

Only 1 mount point is being used out of 4 . Other 3 mount point or paths or directories , mentioned in log.dirs are not been used.

How can i make all the mount / directories , get used.

Thanks

JJ

3 REPLIES 3

Re: How to choose number of kafka partitions

Mentor

@Jacqualin jasmin

Can you locate the kafka server.properties ? The log dir parameter which is a comma separeted should list of directories is where kafka stores the log files. see example below of 3 different log dirs

log.dirs=/xxx/xxx/xxx,/bbb/bbb/bbb,/fff/fff/fff

Ensure that the directories you configure are listed and restart kafka.

Highlighted

Re: How to choose number of kafka partitions

New Contributor

Yes my log.dirs has 4 comma separated directories. Out of 4, only one directory is getting used. How can we make sure all the 4 directories are equally be used.

Right now I have 3 brokers and my topic has 3 partitions. 3 partitions are spread across 3 brokers.

my question is why all the directories were not getting filled ( why only one ). Does the number of partitions of a topic should depend on the number of directories in log.dirs?

Regards

JJ

Re: How to choose number of kafka partitions

Expert Contributor

@Jacqualin jasmin - what is your partitioning scheme? Keep in mind:

The default partitioning strategy has following flow while deciding partition,
  • If a partition is specified in the record, use it
  • If no partition is specified but a key is present choose a partition based on a hash of the key
  • If no partition or key is present choose a partition in a round-robin fashion

(that is from this HCC article)

Is it possible you're pushing records through one partition?

Don't have an account?
Coming from Hortonworks? Activate your account here