Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]
Created 01-21-2014 08:30 AM
Hi Ashok, to increase the file descriptors for the task tracker with a CM controlled cluster you can do the following. Keep in mind that the Alert is only a warning.
Verify your current file descriptor ulimit value to your task tracker processes.
pid=$(ps faux | grep java | grep tasktracker | awk -F " " '{print $2}');cat /proc/$pid/limits;
Perform the following on each node in the cluster. A ulimit of 65536 probably would be enough for most use cases. I just used 262144 as an extreme example.
1. Edit /usr/sbin/cmf-agent and change the ulimit -n setting.
2. Edit /usr/lib64/cmf/service/mapreduce/mapreduce.sh and add
ulimit -n 262144
at the top of the script.
3. Push these script changes out to all the slave nodes
4. A few nodes at a time in the same rack, shutdown datanode and tasktracker services on them.
5. On these stopped nodes run a: service cloudera-scm-agent hard_restart
6. Restart the datanode and tasktracker.
Verify your file descriptor ulimit value to your task tracker processes after making the change.
pid=$(ps faux | grep java | grep tasktracker | awk -F " " '{print $2}');cat /proc/$pid/limits;
-roland