question Re: TaskTracker Max open files (nofile) in Archives of Support Questions (Read Only)

TaskTracker Max open files (nofile)

davidjh — Fri, 16 Sep 2022 08:51:39 GMT

What's the best way to increase Max open files settings for tasktracker process? The agent ignores all settings in the /etc/security/limits.d/*conf files.

Re: TaskTracker Max open files (nofile)

Manikumar Juttukonda — Mon, 23 Dec 2013 20:54:15 GMT

You can try setting the value in /etc/sysctl.conf
fs.epoll.max_user_instances = 2048

Re: TaskTracker Max open files (nofile)

Darren — Thu, 26 Dec 2013 18:59:15 GMT

There isn't a great way to fix this right now. We're looking to fix this for CM 5.

You can edit /usr/lib64/cmf/service/hdfs/hdfs.sh to change your ulimits, but know that this code does not run as root. I think processes will by default get the limits for the root user instead of the ones for the appropriate user, so you can also try changing the ulimits for root.

This issue isn't seen in many clusters, so it could also have something to do with your configuration, how you wrote the map reduce job, or maybe having a lot of small HDFS files. Could also be worth looking into, if only to optimize.

You can find a little more discussion on the topic here:

https://groups.google.com/a/cloudera.org/forum/#!topic/scm-users/G1pRUfYNOWs

Thanks,

Darren

Re: TaskTracker Max open files (nofile)

davidjh — Fri, 17 Jan 2014 23:01:57 GMT

Thanks, but neither of these work. I was eventually able to increase the TaskTracker limits w/

On each node

1. sudo /etc/init.d/cloudera-scm-agent hard_stop
2. Edit parameter in /usr/sbin/cmf-agent script.
3. sudo /etc/init.d/cloudera-scm-agent start

Then restart each tasktracker from Cloudera console.