Community Articles

Find and share helpful community-sourced technical articles.
avatar
Rising Star
Summary

As Kafka works with many log segment files and network connections, the Maximum Process File Descriptors setting may need to be increased in some cases in production deployments, if a broker hosts many partitions

Warnings observed
Concerning : Open file descriptors: 18,646. File descriptor limit: 32,768. 
Percentage in use: 56.90%. Warning threshold: 50.00%.

In case of such open file descriptor warnings in Kafka for CDP, you need to increase file descriptors in Kafka cluster. As per the guide [1], below formula is used

(number of partitions)*(partition size / segment size)

Details about these parameters are as follows:

a. Number of partitions > it can be seen if you go to SMM and brokers tab. Alternatively, you can simply perform a file count of the kafka data directory. You can find this directory in CM > Kafka > log.dirs
b. Partition size > it is the size of log.dirs directory, you can check the disk usage of log.dirs using du -sh command to find this size.
c. Segment size > it can be found by going to CM > Kafka > Segment File Size

Additionally, a simple alternate method of calculation can be to simply search for the number of files in Kafka data directory and set the file descriptors to 2-3 times the value.

 

[1] File descriptor limits

 

1,691 Views
0 Kudos