Reply
New Contributor
Posts: 1
Registered: ‎10-27-2017

Tuning kafka performance to get 1 Million messages/second

Hello,

 

I'm using 3 VM servers, each one has 16 core/ 56 GB Ram /1 TB, to setup a kafka cluster. I work with Kafka 0.10.0 version. I installed a broker on two of them. I have created a topic with 2 partitions, 1 partition/broker and without replication.

 

My goal is to attend 1 000 000 messages / second

 

I made a test with kafka-producer-perf-test.sh script and i get between 150 000 msg/s and 204 000 msg/s.

 

My configuration is:

 

batch size: 8k (8192)
message size: 300 byte (0.3 KB)
thread num: 1

 

The producer configuration:

 

request.required.acks=1
queue.buffering.max.ms=0 #linger.ms=0
compression.codec=none
queue.buffering.max.messages=100000
send.buffer.bytes=100000000

 

Any help will be appreciated to get 1 000 000 msg / s

 

Thank you

Announcements