Created on 12-04-201905:54 AM - edited on 12-09-201912:42 AM by ask_bill_brooks
Knowing the performance of Kafka in general or on your hardware is an important part of capacity planning. Sizing can be hard to calculate, with different message sizes, retention periods, partitions, replica factor, network speeds and even synchronous vs asynchronous to pick from. So many decisions to make, but what type of performance can you realistically expect ?
Benchmarking for Kafka has been done before, however this is a couple of years out of date. There are newer versions of Kafka, newer hardware, faster networking all which have improved performance by many factors. But how ? This article will run the same tests as before with the latest and greatest to see what the improvements are.
3 of the 6 machines will participate as Kafka brokers in a single cluster, and the remaining three nodes will be producers / consumers. Given that we have 10GB/s networking, we should be able to push almost 30GB/s across the three Kafka nodes, and 60GB/s across all the nodes.
To make the testing simpler, I've set some environment variables to re-use in running each command:
Kafka ships with two handy scripts you can use to test your cluster, kafka-producer-perf-test and kafka-consumer-perf-test
Single producer, no consumer, no replication
1 628 081 msg/sec (155.27 MB/sec)
Single producer, no consumer, 3x async replication
1 463 136 msg/sec (140.46 MB/sec)
Single producer, no consumer, 3x syncronous replication
1 226 439 msg/sec (125.11 MB/sec)
3 producers, no consumer, 3 asynchronous replication
3 960 110 msg/sec (377.69 MB/sec)
No producer, single consumer
4 096 100 msg/sec (390.63 MB/sec)
No producer, three consumers
11 813 321 msg/sec (1125 MB/sec)
At this point we've duplicated the testing with the original tests done, hopefully with much improved numbers. However, as the tests were done with 100 byte records, the tests were re-run with 7KB records and optimised Kafka settings (larger Heap size at 8GB, larger batch sizes and some snappy compression applied).