Created on 06-28-202312:47 AM - edited on 06-29-202310:39 PM by VidyaSargur
As Kafka works with many log segment files and network connections, the Maximum Process File Descriptors setting may need to be increased in some cases in production deployments, if a broker hosts many partitions
Concerning : Open file descriptors: 18,646. File descriptor limit: 32,768. Percentage in use: 56.90%. Warning threshold: 50.00%.
In case of such open file descriptor warnings in Kafka for CDP, you need to increase file descriptors in Kafka cluster. As per the guide , below formula is used
(number of partitions)*(partition size / segment size)
Details about these parameters are as follows:
a. Number of partitions > it can be seen if you go to SMM and brokers tab. Alternatively, you can simply perform a file count of the kafka data directory. You can find this directory in CM > Kafka > log.dirs b. Partition size > it is the size of log.dirs directory, you can check the disk usage of log.dirs using du -sh command to find this size. c. Segment size > it can be found by going to CM > Kafka > Segment File Size
Additionally, a simple alternate method of calculation can be to simply search for the number of files in Kafka data directory and set the file descriptors to 2-3 times the value.