@Boris
Demerov
Yes. Reference
provided by @Randy Gelhausen is
awesome, one of the best articles in HCC. It covers Kafka tuning practices
beyond the scope of your question, but it is a must read article.
I'd like to point out a few good rules of thumb related to your
question, which Wes in his article covered too (I extracted and commented a few):
Set num.io.threads to at
least no. of disks you are going to use by default its 8. It be can higher than
the number of disks.
A common broker server
has 8 disks. That is my current experience, however, this number can be
increased.
Set num.network.threads
higher based on number of concurrent producers, consumers, and replication
factor.
The default value of 3
has been set based on field experience, however, you can take an iterative
approach and test different values until you find what is optimal for your
case.
Ideally you want to
assign the default number of partitions (num.partitions) to at least n-1
servers. This can break up the write workload and it allows for greater
parallelism on the consumer side. Remember that Kafka does total ordering
within a partition, not over multiple partitions, so make sure you partition
intelligently on the producer side to parcel up units of work that might span
multiple messages/events.
Consumers
benefit from this approach, on producers – careful design is recommended. You
need to balance the benefits between producer and consumers based on your
business needs.
Kafka is designed for
small messages. I recommend you to avoid using kafka for larger messages. If
that’s not avoidable there are several ways to go about sending larger messages
like 1MB. Use compression if the original message is json, xml or text using
compression is the best option to reduce the size. Large messages will affect
your performance and throughput. Check your topic partitions and
replica.fetch.size to make sure it doesn’t go over your physical ram. Another
approach is to break the message into smaller chunks and use the same message
key to send it same partition. This way you are sending small messages and these
can be re-assembled at the consumer side.
This complicates your
Producer and Consumer code in case of very large messages. Design carefully how
Producers and Consumers deal with large size messages. There are many ways to
implement compression or chunking, as well decompression and assembly. Choose
after testing your approach. For example, a high compression ratio is most of
the time an advantage but it comes with a price paid for
compression/decompression time. It is sometimes more efficient to have a lesser
compression as long as you can reduce the size of the message under 1MB, but
faster compression/decompression. It all comes-down to your SLAs whether they
are ms or seconds.
I am sure I did not cover
the domain completely, but hopefully this helps.
