Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (1)
Cloudera Employee

Introduction

Knowing the performance of Kafka in general or on your hardware is an important part of capacity planning. Sizing can be hard to calculate, with different message sizes, retention periods, partitions, replica factor, network speeds and even synchronous vs asynchronous to pick from. So many decisions to make, but what type of performance can you realistically expect ?

 

Benchmarking for Kafka has been done before, however this is a couple of years out of date. There are newer versions of Kafka, newer hardware, faster networking all which have improved performance by many factors. But how ? This article will run the same tests as before with the latest and greatest to see what the improvements are.

 

Hardware specification:

Cloud Provider AWS using m5.4xlarge * 6 machines
OS Centos 7
CPU 16 cores
RAM 64GB
Disk 50GB OS, 6 * 20 GB SSD IO Optimised for Kafka log dirs
Networking 10GB/s
Kafka Version CDH 6.3 with Kafka 2.3
Message size 100 bytes per message

 

3 of the 6 machines will participate as Kafka brokers in a single cluster, and the remaining three nodes will be producers / consumers. Given that we have 10GB/s networking, we should be able to push almost 30GB/s across the three Kafka nodes, and 60GB/s across all the nodes.

 

To make the testing simpler, I've set some environment variables to re-use in running each command:

 

 

BOOTSTRAP=10.0.0.4:9092,10.0.0.5:9092,10.0.0.6:9092

 

 

 

First we need to create the various topics, depending on the partition count and replicas:

 

 

kafka-topics --bootstrap-server ${BOOTSTRAP} --create --topic test-rep-one --partitions 6 --replication-factor 1
kafka-topics --bootstrap-server ${BOOTSTRAP} --create --topic test-rep-three --partitions 6 --replication-factor 3
kafka-topics --bootstrap-server ${BOOTSTRAP} --create --topic test-7k --partitions 18 --replication-factor 3

 

 

 

Kafka ships with two handy scripts you can use to test your cluster, kafka-producer-perf-test and kafka-consumer-perf-test

 

Test results

Single producer, no consumer, no replication 1 628 081 msg/sec (155.27 MB/sec)
Single producer, no consumer, 3x async replication 1 463 136 msg/sec (140.46 MB/sec)
Single producer, no consumer, 3x syncronous replication 1 226 439 msg/sec (125.11 MB/sec)
3 producers, no consumer, 3 asynchronous replication 3 960 110 msg/sec (377.69 MB/sec)
No producer, single consumer 4 096 100 msg/sec (390.63 MB/sec)
No producer, three consumers 11 813 321 msg/sec (1125 MB/sec)

 

 At this point we've duplicated the testing with the original tests done, hopefully with much improved numbers.
However, as the tests were done with 100 byte records, the tests were re-run with 7KB records and optimised Kafka settings (larger Heap size at 8GB, larger batch sizes and some snappy compression applied).

 

Optimised Kafka results, 7k records

6 producers, no consumers, 3x async replication (larger batch sizes, snappy compression) 1 070 970 msg/sec (7321 MB/sec)
0 producers, 6 consumers 963 071 msg/sec (6896 MB/sec)

 

Over 1 million messages a second, reading and writing 7kb per message. We've reached the networking limit!

 

The commands used for each test if you would like to reproduce yourself:

 

 

Test 1:
kafka-producer-perf-test --topic test-rep-one --num-records 50000000 --record-size 100 --throughput -1 --producer-props acks=0 bootstrap.servers=${BOOTSTRAP} 

Test 2:
kafka-producer-perf-test --topic test-rep-three --num-records 50000000 --record-size 100 --throughput -1 --producer-props acks=0 bootstrap.servers=${BOOTSTRAP}

Test 3:
kafka-producer-perf-test --topic test-rep-three --num-records 50000000 --record-size 100 --throughput -1 --producer-props acks=1 bootstrap.servers=${BOOTSTRAP}

Test 4 (run three instances in parallel, one on each node):
kafka-producer-perf-test --topic test-rep-three --num-records 50000000 --record-size 100 --throughput -1 --producer-props acks=0 bootstrap.servers=${BOOTSTRAP}

Test 5:
kafka-consumer-perf-test --broker-list ${BOOTSTRAP} --messages 50000000 --topic test-rep-three --threads 1 --timeout 60000 --print-metrics --num-fetch-threads 6

Test 6 (run three instances in parallel, one on each node):
kafka-consumer-perf-test --broker-list ${BOOTSTRAP} --messages 50000000 --topic test-rep-three --threads 1 --timeout 60000 --print-metrics --num-fetch-threads 6

Test 7 (run a producer on each node, including the Kafka brokers):
kafka-producer-perf-test --topic test-7k --num-records 50000000 --record-size 7168 --throughput -1 --producer-props acks=0 bootstrap.servers=${BOOTSTRAP} linger.ms=100 compression.type=snappy

Test 8 (run a consumer on each node, including the Kafka brokers):
kafka-consumer-perf-test --broker-list ${BOOTSTRAP} --messages 50000000 --topic test-7k --threads 1 --timeout 60000 --print-metrics --num-fetch-threads 18

 

 

937 Views
Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
4 of 4
Last update:
‎12-09-2019 12:42 AM
Updated by:
 
Top Kudoed Authors