Support Questions

Find answers, ask questions, and share your expertise
Celebrating as our community reaches 100,000 members! Thank you!

how to increase open files on kafka cluster


We have 10 kafka machines with kafka version - 1.X

this kafka cluster version is part of HDP version - 2.6.5

We noticed that under /var/log/kafka/server.log the following message


ERROR Error while accepting connection {} Too many open files


We saw also additionally


Broker 21 stopped fetcher for partition ...................... because they are in the failed log dir /kafka/kafka-logs {kafka.server.ReplicaManager}




WARN Received a PartitionLeaderEpoch assignment for an epoch < latestEpoch. this implies messages have arrived out of order. New: {epoch:0, offset:2227488}, Currnet: {epoch 2, offset:261} for Partition: cars-list-75 {kafka.server.epochLeaderEpocHFileCache}


so regarding to the issue -


ERROR Error while accepting connection {} Too many open files



how to increase the MAX open files , in order to avoid this issue



Expert Contributor




 Kafka broker needs at least the following number of file descriptors to just track log segment files:



(number of partitions)*(partition size / segment size)



You can review the current limits configuration under:



cat /proc/<kafka_pid>/limits



If you want to change them, if you're using ambari console you can go to > Kafka > config > and search for "kafka_user_nofile_limit"


Finally, To see open file descriptors, run: