Open file descriptors issue on Kudu

Question

Hello,We have been getting alerts for Open file descriptors on Kudu T-servers “Concerning: Open file descriptors: 16,954. File descriptor limit: 32,768. Percentage in use: 51.74%. Warning threshold: 50.00%.” request assistance in resolving this. Also does restarting services on  affected T-Servers have any performance impact on Kudu as whole.[root@myserver ~]# ulimit -acore file size          (blocks, -c) 0data seg size           (kbytes, -d) unlimitedscheduling priority             (-e) 0file size               (blocks, -f) unlimitedpending signals                 (-i) 1546606max locked memory       (kbytes, -l) 64max memory size         (kbytes, -m) unlimitedopen files                      (-n) 1024pipe size            (512 bytes, -p) 8POSIX message queues     (bytes, -q) 819200real-time priority              (-r) 0stack size              (kbytes, -s) 8192cpu time               (seconds, -t) unlimitedmax user processes              (-u) 1546606virtual memory          (kbytes, -v) unlimitedfile locks                      (-x) unlimited [root@myserver ~]# cat /proc/sys/fs/file-max39233193 [root@myserver ~]# cat /proc/sys/fs/file-nr35200   0       39233193 [root@myserver ~]#  lsof -u kudu |wc -l18326 RegardsAmn

kingpin · Accepted Answer

@Amn_468 , Depends if you are receiving the file descriptor warning message across all the TS and if you happen to change the fds threshold across kudu Tserver roles then you should be doing in non prod hours or ensure no jobs are running. Please be informed that kudu doesnt offer rolling restart feature.

Service level config TS

CM->kudu->config->file descriptor for TS-> require all TS restart

However if the warning appears only for one specific TS and if you plan to change the thershold for one specific TS then it shouldnt cause any issue as its a distributed system and one worker role restart wont impact on your ongoing operations.

CM->kudu->instances->select the TS instance which throws the warning-> config-> set the fds value ->restart specific TS

kingpin · Answer

Hi @AM_01 ,

The alert regarding fd is a warning message that you have crossed the warning threshold of open fds for kudu.

Ideally there are 3 sources which hold up the fds in kudu.
1. File cache 2. Hot replica 3. cold replica

Typically there is no downside of increasing the fds in kudu tserver. You can increase the fds from CM->kudu->config->file descriptor ->increase the value to 64k or you may also tune the threshold and set the warning to 75%.

Restart of TS is required if you tune the file descriptor.

Hope this helps!!

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.