Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

kafka bandwidth turning

kafka bandwidth turning

Expert Contributor

I use kafka. the server has 1G ethernet

recent i found that then lan bandwidth of machines is too high, i use some tools to monitor the traffic and foud that too many trasfer usage of kafka. total bandwidth frequently reach 400mbps, and ping of other node can from 0.1 ms suddenly to 30ms....


So is there some way to turning the bandwidth usage??

3 REPLIES 3

Re: kafka bandwidth turning

Mentor

@darkz yu

Here is a good reference for Kafka tuning .Kafka Performance tuning and ways for Kafka Optimization

Hope that helps

Re: kafka bandwidth turning

Expert Contributor

I just want to know which option to turning can reduce bandwidth usage


Re: kafka bandwidth turning

Mentor

@darkz yu

I don't think there is a silver bullet because you have to balance the producers, brokers, and consumers but I would think the most important are these 2 parameters below

batch.size = 16384 # default size 

Smaller batch size will make batching less common and may reduce throughput a batch size of zero will disable batching entirely. A very large batch size may use memory a bit more wastefully as we will always allocate a buffer of the specified batch size in anticipation of additional records.

linger.ms = 0 #default 

linger.ms=5 would have the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absence of a load.