Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

AmbariAgent on RHEL6 low ulimits after boot which propagate to HDFS

Highlighted

AmbariAgent on RHEL6 low ulimits after boot which propagate to HDFS

Contributor

We had a problem on Datanodes, which had low nr open files limits. This will lead to exceptions when heavily using hdfs.

After a machine reboot the service ambari-agent starts automatically. This is a sub-process from the init process. The limits from the init process is 1024 (soft) / 4096 (hard). Any fork from this process by default will copy the limits. If AmbariAgent is running as a root user the startup command does not change to another user and not use the limits as configured in /etc/security/limits.conf & /etc/security/limits.d/*.conf

# Print AmbariAgent, HDFS and YARN limits on a DataNode
grep 'open files' /proc/$(ps aux | grep "[A]mbariAgent" |  awk '{print $2}')/limits
grep 'open files' /proc/$(pgrep -U hdfs)/limits
grep 'open files' /proc/$(pgrep -u yarn java)/limits

If you restart HDFS from the AmbariServer UI the limits are still low. After restarting the AmbariAgent service the limits of AmbariAgent are increased. When restart HDFS after AmbariAgent the limits are correct. The problem still exists for nodes that are rebooted.

Our Solution:

Change the /etc/init.d/ambari-agent script on all the nodes and add a line just before starting the ambari agent:

case "$1" in
  start)
        # Start is a 'fork' of init process and had 1024/4048 limits!
        CURRENT_HARD_LIMIT=$(ulimit -Hn)
        if [[ $CURRENT_HARD_LIMIT -lt 5000 ]]; then
            ulimit -Hn 128000
        fi
        $command_prefx "/usr/sbin/ambari-agent $@"
        ;;
  stop)
4 REPLIES 4
Highlighted

Re: AmbariAgent on RHEL6 low ulimits after boot which propagate to HDFS

@Alexander Bij

Try to add in /etc/security/limits.d/hdfs.conf

hdfs - nofile 128000

hdfs -nproc 65536

and then data node restart.

Highlighted

Re: AmbariAgent on RHEL6 low ulimits after boot which propagate to HDFS

Contributor

Thx for the reply. I forgot to mention that these settings that you suggest are also present. unfortunately they are not applied after a server reboot when Ambari starts automatically.

Highlighted

Re: AmbariAgent on RHEL6 low ulimits after boot which propagate to HDFS

@Alexannder Bij,

If this is help you to resolve your issue, then lets close this one.

Re: AmbariAgent on RHEL6 low ulimits after boot which propagate to HDFS

have you check with hdfs user ?

hdfs - nofile 128000

hdfs - nproc 65536

Run command with hdfs user

ulimit -n

Don't have an account?
Coming from Hortonworks? Activate your account here