Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How can i improuve kafka performance

Highlighted

How can i improuve kafka performance

Explorer

Hello,

i am using 3 virtual server (16 core / 56 Go Ram / 1 TO ) to setup a kafka cluster. i installed a broker on two of them. i have created a topic with two partitions.

my objectif is to attend 1 000 000 messages / second. until now i am between 150 000 m/s and 204 000 m/s

batch size: 8k (8192)

message size: 300b (0.3 ko)

compression codec: none

thread num: 1

please find bellow producer configuration:

request.required.acks=1

queue.buffering.max.ms=0
compression.codec=none
queue.buffering.max.messages=100000
send.buffer.bytes=100000000

Any help will be appreciated to get 1 000 000 m / s

thank you

1 REPLY 1

Re: How can i improuve kafka performance

Rising Star

Here's a great article on Kafka performance tuning: https://community.hortonworks.com/articles/80813/kafka-best-practices-1.html

Don't have an account?
Coming from Hortonworks? Activate your account here