Ulimit settings not respected when service is started from Ambari

[#1]

SYMPTOMS: Ambari configures "hdfs - nofile hdfs - nofile 128000" in /etc/security/limits.d/hdfs.conf but when the datanode (or any other) process is started by Ambari, it still runs with a limit of only 8129 open files:

sudo grep open /proc/19608/limits 
Max open files 8192 8192 files 

(19608 is the pid of the datanode process in this case).

As a consequence we get millions of entries in datanode logs resulting in a lot of disk / IO bottleneck.

WARN mortbay.log (Slf4jLog.java:warn(89)) - 
EXCEPTION java.io.IOException: Too many open files 

ROOT CAUSE: Ambari should check uncommented "session required pam_limits.so" line in /etc/pam.d/su and /etc/pam.d/sudo , otherwise ulimit values from /etc/security/limits.d/hdfs.conf (+ yarn.conf, hive.conf, ams.conf) won't be applied. This is a known issue reported in internal BUG-38892.

SOLUTION: Not yet fixed.

WORKAROUND: Uncomment the line "session required pam_limits.so " in /etc/pam.d/su on each node and restart services.Log a case with HWX support team to get a patch for the bug.

[#2]

Thank you! I've pulled most of my hair out researching this problem. I'm thankful you posted a workaround!

[#3]

Here is my solution

https://community.hortonworks.com/questions/208928/increase-open-file-limit-of-the-user-to-scale-for...