Open file descriptors issue on Kudu

[#1]

Hello,

We have been getting alerts for Open file descriptors on Kudu T-servers “Concerning: Open file descriptors: 16,954. File descriptor limit: 32,768. Percentage in use: 51.74%. Warning threshold: 50.00%.” request assistance in resolving this. Also does restarting services on  affected T-Servers have any performance impact on Kudu as whole.

[root@myserver ~]# ulimit -a

core file size          (blocks, -c) 0

data seg size           (kbytes, -d) unlimited

scheduling priority             (-e) 0

file size               (blocks, -f) unlimited

pending signals                 (-i) 1546606

max locked memory       (kbytes, -l) 64

max memory size         (kbytes, -m) unlimited

open files                      (-n) 1024

pipe size            (512 bytes, -p) 8

POSIX message queues     (bytes, -q) 819200

real-time priority              (-r) 0

stack size              (kbytes, -s) 8192

cpu time               (seconds, -t) unlimited

max user processes              (-u) 1546606

virtual memory          (kbytes, -v) unlimited

file locks                      (-x) unlimited

[root@myserver ~]# cat /proc/sys/fs/file-max

39233193

[root@myserver ~]# cat /proc/sys/fs/file-nr

35200   0       39233193

[root@myserver ~]#  lsof -u kudu |wc -l

18326

Regards

Amn

[#2]

Hi @AM ,

The alert regarding fd is a warning message that you have crossed the warning threshold of open fds for kudu. 

Ideally there are 3 sources which hold up the fds in kudu. 
1. File cache 2. Hot replica  3. cold replica

Typically there is no downside of increasing the fds in kudu tserver. You can increase the fds from CM->kudu->config->file descriptor ->increase the value to 64k or you may also tune the threshold and set the warning to 75%.

Restart of TS is required if you tune the file descriptor. 

Hope this helps!!

[#3]

Hi @kingpin 

Thanks for the assistance, wanted to confirm if restarting TS have any impact on Kudu service?

Regards

[#4 · Accepted solution]

@Amn_468 , Depends if you are receiving the file descriptor warning message across all the TS and if you happen to change the fds threshold across kudu Tserver roles then you should be doing in non prod hours or ensure no jobs are running. Please be informed that kudu doesnt offer rolling restart feature.

Service level config TS

CM->kudu->config->file descriptor for TS-> require all TS restart

However if the warning appears only for one specific TS and if you plan to change the thershold for one specific TS then it shouldnt cause any issue as its a distributed system and one worker role restart wont impact on your ongoing operations.

CM->kudu->instances->select the TS instance which throws the warning-> config-> set the fds value ->restart specific TS

[#5]

Hi @kingpin 

Appreciate all your assistance 

Regards

Amn