We are facing issues with Open File descriptors in Task tracker from cloudera manager and most of the Task tracker are in critical health. Cloudera Manager: ver 4.7 Please find below the status of one such Task Tracker: "Open file descriptors: 24,241. File descriptor limit: 32,768. Percentage in use: 73.98%. Critical threshold: 70.00%." we have checked the File descriptor with the OS using the command "ulimit -a" and it shows "unlimited", core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 118683 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 1024 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited In Cloudera documentation, we see the description for TaskTracker File Descriptors as: "The health check thresholds of the number of file descriptors used. A failure of this health check may indicate a bug in either Hadoop or Cloudera Manager. Contact Cloudera support." We found a similar issue in the issue list, but not sure whether they are talking about the same underlying problem: https://issues.cloudera.org/browse/DISTRO-530 Is this indeed a bug in the underlying framework or is there a way to increase the Open File Descriptors thorugh Cloudera Manager? Any other approach to recover from this issue is also appreciated. Thanks
... View more
