Thank you very much for reading my question. We have a hadoop cluster with 4 brokers installed and 7 topics currently in production state. Since some days ago, there is a need to restart kafka brokers every few hours because we see that producers performance decrease a lot, and the time needed to insert a message in kafka grow from some miliseconds to seconds...
Consumers seem to be alright.
Our problem is that we don't know what is happening and what parameters to follow to resolve this problem. Where should we start for diagnosing ? Today we know that restarting a broker every 7 or 8 minutes resolve the problem for some time, but we don´t know why.
Can anybody help us?
Thank you very much,
Increase the kafka open files no.
kafka - nofile 98304
and restart the kafka.
Please check link, if this help you
And accept the answer if this is resolved your issue.
Thank you very much. I have modified the parameter but I have pending a kafka reboot. Once I do it I'll tell you.