Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Need to restart brokers every few hours because of performance decrease

Highlighted

Need to restart brokers every few hours because of performance decrease

Hello,

Thank you very much for reading my question. We have a hadoop cluster with 4 brokers installed and 7 topics currently in production state. Since some days ago, there is a need to restart kafka brokers every few hours because we see that producers performance decrease a lot, and the time needed to insert a message in kafka grow from some miliseconds to seconds...

Consumers seem to be alright.

Our problem is that we don't know what is happening and what parameters to follow to resolve this problem. Where should we start for diagnosing ? Today we know that restarting a broker every 7 or 8 minutes resolve the problem for some time, but we don´t know why.

Can anybody help us?

Thank you very much,

Silvio

4 REPLIES 4
Highlighted

Re: Need to restart brokers every few hours because of performance decrease

@Silvio

Try this,

Increase the kafka open files no.

/etc/security/limits.conf:

kafka - nofile 98304

and restart the kafka.

Please check link, if this help you

http://www.bectortapu.com/kafka-performance-tuning/

And accept the answer if this is resolved your issue.

Highlighted

Re: Need to restart brokers every few hours because of performance decrease

Thank you very much. I have modified the parameter but I have pending a kafka reboot. Once I do it I'll tell you.

Thanks again

Highlighted

Re: Need to restart brokers every few hours because of performance decrease

Welcome and let me know if issue is resolved.

Highlighted

Re: Need to restart brokers every few hours because of performance decrease

Super Guru

How large are the queues?

How much memory on each node?

read this: https://community.hortonworks.com/articles/550/unofficial-storm-and-kafka-best-practices-guide.html

Don't have an account?
Coming from Hortonworks? Activate your account here