Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

nodemanager start failed: ulimit: open files: cannot modify limit: Operation not permitted

nodemanager start failed: ulimit: open files: cannot modify limit: Operation not permitted

New Contributor

ambari-server version:Version1.7.0,with HDP-2.2 repository.

namenode start failed with error message:

stderr: 
2016-03-31 16:03:11,470 - Error while executing command 'start':
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 123, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/YARN/package/scripts/nodemanager.py", line 42, in start
    action='start'
  File "/data/log/ambari-agent-1.0/cache/stacks/HDP/2.0.6/services/YARN/package/scripts/service.py", line 59, in service
    initial_wait=5
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 149, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 115, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 241, in action_run
    raise ex
Fail: Execution of 'ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1' returned 1. /etc/profile: line 81: ulimit: open files: cannot modify limit: Operation not permitted
 stdout:
2016-03-31 16:03:02,574 - Group['hadoop'] {'ignore_failures': False}
2016-03-31 16:03:02,575 - Modifying group hadoop
2016-03-31 16:03:02,589 - Group['nobody'] {'ignore_failures': False}
2016-03-31 16:03:02,590 - Modifying group nobody
2016-03-31 16:03:02,604 - Group['users'] {'ignore_failures': False}
2016-03-31 16:03:02,604 - Modifying group users
2016-03-31 16:03:02,618 - Group['nagios'] {'ignore_failures': False}
2016-03-31 16:03:02,619 - Modifying group nagios
2016-03-31 16:03:02,633 - User['nobody'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'nobody']}
2016-03-31 16:03:02,633 - Modifying user nobody
2016-03-31 16:03:02,646 - User['mapred'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2016-03-31 16:03:02,647 - Modifying user mapred
2016-03-31 16:03:02,660 - User['nagios'] {'gid': 'nagios', 'ignore_failures': False, 'groups': [u'hadoop']}
2016-03-31 16:03:02,660 - Modifying user nagios
2016-03-31 16:03:02,673 - User['ambari-qa'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'users']}
2016-03-31 16:03:02,673 - Modifying user ambari-qa
2016-03-31 16:03:02,686 - User['flume'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2016-03-31 16:03:02,686 - Modifying user flume
2016-03-31 16:03:02,699 - User['zookeeper'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2016-03-31 16:03:02,699 - Modifying user zookeeper
2016-03-31 16:03:02,712 - User['hdfs'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2016-03-31 16:03:02,713 - Modifying user hdfs
2016-03-31 16:03:02,725 - User['kafka'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2016-03-31 16:03:02,726 - Modifying user kafka
2016-03-31 16:03:02,739 - User['yarn'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': [u'hadoop']}
2016-03-31 16:03:02,739 - Modifying user yarn
2016-03-31 16:03:02,752 - File['/var/lib/ambari-agent/data/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2016-03-31 16:03:02,753 - Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 2>/dev/null'] {'not_if': 'test $(id -u ambari-qa) -gt 1000'}
2016-03-31 16:03:02,764 - Skipping Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 2>/dev/null'] due to not_if
2016-03-31 16:03:02,764 - Directory['/etc/hadoop/conf.empty'] {'owner': 'root', 'group': 'root', 'recursive': True}
2016-03-31 16:03:02,764 - Link['/etc/hadoop/conf'] {'not_if': 'ls /etc/hadoop/conf', 'to': '/etc/hadoop/conf.empty'}
2016-03-31 16:03:02,774 - Skipping Link['/etc/hadoop/conf'] due to not_if
2016-03-31 16:03:02,783 - File['/etc/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs'}
2016-03-31 16:03:02,792 - Execute['/bin/echo 0 > /selinux/enforce'] {'only_if': 'test -f /selinux/enforce'}
2016-03-31 16:03:02,802 - Skipping Execute['/bin/echo 0 > /selinux/enforce'] due to only_if
2016-03-31 16:03:02,802 - Directory['/var/log/hadoop'] {'owner': 'root', 'group': 'hadoop', 'mode': 0775, 'recursive': True}
2016-03-31 16:03:02,803 - Directory['/var/run/hadoop'] {'owner': 'root', 'group': 'root', 'recursive': True}
2016-03-31 16:03:02,804 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'recursive': True}
2016-03-31 16:03:02,807 - File['/etc/hadoop/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2016-03-31 16:03:02,809 - File['/etc/hadoop/conf/health_check'] {'content': Template('health_check-v2.j2'), 'owner': 'hdfs'}
2016-03-31 16:03:02,809 - File['/etc/hadoop/conf/log4j.properties'] {'content': '...', 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2016-03-31 16:03:02,813 - File['/etc/hadoop/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs'}
2016-03-31 16:03:02,813 - File['/etc/hadoop/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2016-03-31 16:03:02,926 - HdfsDirectory['/app-logs'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '/usr/bin/kinit', 'mode': 0777, 'owner': 'yarn', 'group': 'hadoop', 'action': ['create_delayed'], 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'recursive_chmod': True}
2016-03-31 16:03:02,926 - HdfsDirectory['/mapred'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '/usr/bin/kinit', 'owner': 'mapred', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create_delayed']}
2016-03-31 16:03:02,927 - HdfsDirectory['/mapred/system'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '/usr/bin/kinit', 'owner': 'hdfs', 'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'action': ['create_delayed']}
2016-03-31 16:03:02,927 - HdfsDirectory['/mr-history/tmp'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '/usr/bin/kinit', 'mode': 0777, 'owner': 'mapred', 'group': 'hadoop', 'action': ['create_delayed'], 'bin_dir': '/usr/hdp/current/hadoop-client/bin'}
2016-03-31 16:03:02,927 - HdfsDirectory['/mr-history/done'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '/usr/bin/kinit', 'mode': 01777, 'owner': 'mapred', 'group': 'hadoop', 'action': ['create_delayed'], 'bin_dir': '/usr/hdp/current/hadoop-client/bin'}
2016-03-31 16:03:02,927 - HdfsDirectory['None'] {'security_enabled': False, 'keytab': [EMPTY], 'conf_dir': '/etc/hadoop/conf', 'hdfs_user': 'hdfs', 'kinit_path_local': '/usr/bin/kinit', 'action': ['create'], 'bin_dir': '/usr/hdp/current/hadoop-client/bin'}
2016-03-31 16:03:02,929 - Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` /app-logs /mapred /mapred/system /mr-history/tmp /mr-history/done && hadoop --config /etc/hadoop/conf fs -chmod -R 777 /app-logs && hadoop --config /etc/hadoop/conf fs -chmod  777 /mr-history/tmp && hadoop --config /etc/hadoop/conf fs -chmod  1777 /mr-history/done && hadoop --config /etc/hadoop/conf fs -chown  mapred /mapred && hadoop --config /etc/hadoop/conf fs -chown  hdfs /mapred/system && hadoop --config /etc/hadoop/conf fs -chown  yarn:hadoop /app-logs && hadoop --config /etc/hadoop/conf fs -chown  mapred:hadoop /mr-history/tmp /mr-history/done'] {'not_if': "su - hdfs -c 'export PATH=$PATH:/usr/hdp/current/hadoop-client/bin ; hadoop --config /etc/hadoop/conf fs -ls /app-logs /mapred /mapred/system /mr-history/tmp /mr-history/done'", 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
2016-03-31 16:03:05,288 - Skipping Execute['hadoop --config /etc/hadoop/conf fs -mkdir `rpm -q hadoop | grep -q "hadoop-1" || echo "-p"` /app-logs /mapred /mapred/system /mr-history/tmp /mr-history/done && hadoop --config /etc/hadoop/conf fs -chmod -R 777 /app-logs && hadoop --config /etc/hadoop/conf fs -chmod  777 /mr-history/tmp && hadoop --config /etc/hadoop/conf fs -chmod  1777 /mr-history/done && hadoop --config /etc/hadoop/conf fs -chown  mapred /mapred && hadoop --config /etc/hadoop/conf fs -chown  hdfs /mapred/system && hadoop --config /etc/hadoop/conf fs -chown  yarn:hadoop /app-logs && hadoop --config /etc/hadoop/conf fs -chown  mapred:hadoop /mr-history/tmp /mr-history/done'] due to not_if
2016-03-31 16:03:05,288 - Directory['/data/hadoop/yarn/local'] {'owner': 'yarn', 'ignore_failures': True, 'recursive': True}
2016-03-31 16:03:05,289 - Directory['/data/hadoop/yarn/log'] {'owner': 'yarn', 'ignore_failures': True, 'recursive': True}
2016-03-31 16:03:05,289 - Directory['/var/run/hadoop-yarn/yarn'] {'owner': 'yarn', 'group': 'hadoop', 'recursive': True}
2016-03-31 16:03:05,290 - Directory['/var/log/hadoop-yarn/yarn'] {'owner': 'yarn', 'group': 'hadoop', 'recursive': True}
2016-03-31 16:03:05,290 - Directory['/var/run/hadoop-mapreduce/mapred'] {'owner': 'mapred', 'group': 'hadoop', 'recursive': True}
2016-03-31 16:03:05,290 - Directory['/var/log/hadoop-mapreduce/mapred'] {'owner': 'mapred', 'group': 'hadoop', 'recursive': True}
2016-03-31 16:03:05,290 - Directory['/var/log/hadoop-yarn'] {'owner': 'yarn', 'ignore_failures': True, 'recursive': True}
2016-03-31 16:03:05,291 - XmlConfig['core-site.xml'] {'group': 'hadoop', 'conf_dir': '/etc/hadoop/conf', 'mode': 0644, 'configuration_attributes': ..., 'owner': 'hdfs', 'configurations': ...}
2016-03-31 16:03:05,301 - Generating config: /etc/hadoop/conf/core-site.xml
2016-03-31 16:03:05,301 - File['/etc/hadoop/conf/core-site.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2016-03-31 16:03:05,302 - Writing File['/etc/hadoop/conf/core-site.xml'] because contents don't match
2016-03-31 16:03:05,302 - XmlConfig['mapred-site.xml'] {'group': 'hadoop', 'conf_dir': '/etc/hadoop/conf', 'mode': 0644, 'configuration_attributes': ..., 'owner': 'yarn', 'configurations': ...}
2016-03-31 16:03:05,309 - Generating config: /etc/hadoop/conf/mapred-site.xml
2016-03-31 16:03:05,310 - File['/etc/hadoop/conf/mapred-site.xml'] {'owner': 'yarn', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2016-03-31 16:03:05,311 - Writing File['/etc/hadoop/conf/mapred-site.xml'] because contents don't match
2016-03-31 16:03:05,311 - Changing owner for /etc/hadoop/conf/mapred-site.xml from 513 to yarn
2016-03-31 16:03:05,311 - XmlConfig['yarn-site.xml'] {'group': 'hadoop', 'conf_dir': '/etc/hadoop/conf', 'mode': 0644, 'configuration_attributes': ..., 'owner': 'yarn', 'configurations': ...}
2016-03-31 16:03:05,318 - Generating config: /etc/hadoop/conf/yarn-site.xml
2016-03-31 16:03:05,318 - File['/etc/hadoop/conf/yarn-site.xml'] {'owner': 'yarn', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2016-03-31 16:03:05,320 - Writing File['/etc/hadoop/conf/yarn-site.xml'] because contents don't match
2016-03-31 16:03:05,321 - XmlConfig['capacity-scheduler.xml'] {'group': 'hadoop', 'conf_dir': '/etc/hadoop/conf', 'mode': 0644, 'configuration_attributes': ..., 'owner': 'yarn', 'configurations': ...}
2016-03-31 16:03:05,328 - Generating config: /etc/hadoop/conf/capacity-scheduler.xml
2016-03-31 16:03:05,328 - File['/etc/hadoop/conf/capacity-scheduler.xml'] {'owner': 'yarn', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0644, 'encoding': 'UTF-8'}
2016-03-31 16:03:05,329 - Writing File['/etc/hadoop/conf/capacity-scheduler.xml'] because contents don't match
2016-03-31 16:03:05,329 - Changing owner for /etc/hadoop/conf/capacity-scheduler.xml from 518 to yarn
2016-03-31 16:03:05,329 - File['/etc/hadoop/conf/yarn.exclude'] {'owner': 'yarn', 'group': 'hadoop'}
2016-03-31 16:03:05,331 - File['/etc/security/limits.d/yarn.conf'] {'content': Template('yarn.conf.j2'), 'mode': 0644}
2016-03-31 16:03:05,333 - File['/etc/security/limits.d/mapreduce.conf'] {'content': Template('mapreduce.conf.j2'), 'mode': 0644}
2016-03-31 16:03:05,336 - File['/etc/hadoop/conf/yarn-env.sh'] {'content': InlineTemplate(...), 'owner': 'yarn', 'group': 'hadoop', 'mode': 0755}
2016-03-31 16:03:05,338 - File['/etc/hadoop/conf/mapred-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs'}
2016-03-31 16:03:05,339 - File['/etc/hadoop/conf/taskcontroller.cfg'] {'content': Template('taskcontroller.cfg.j2'), 'owner': 'hdfs'}
2016-03-31 16:03:05,340 - XmlConfig['mapred-site.xml'] {'owner': 'mapred', 'group': 'hadoop', 'conf_dir': '/etc/hadoop/conf', 'configuration_attributes': ..., 'configurations': ...}
2016-03-31 16:03:05,347 - Generating config: /etc/hadoop/conf/mapred-site.xml
2016-03-31 16:03:05,347 - File['/etc/hadoop/conf/mapred-site.xml'] {'owner': 'mapred', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2016-03-31 16:03:05,348 - Changing owner for /etc/hadoop/conf/mapred-site.xml from 521 to mapred
2016-03-31 16:03:05,348 - XmlConfig['capacity-scheduler.xml'] {'owner': 'hdfs', 'group': 'hadoop', 'conf_dir': '/etc/hadoop/conf', 'configuration_attributes': ..., 'configurations': ...}
2016-03-31 16:03:05,355 - Generating config: /etc/hadoop/conf/capacity-scheduler.xml
2016-03-31 16:03:05,356 - File['/etc/hadoop/conf/capacity-scheduler.xml'] {'owner': 'hdfs', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2016-03-31 16:03:05,356 - Changing owner for /etc/hadoop/conf/capacity-scheduler.xml from 521 to hdfs
2016-03-31 16:03:05,357 - File['/var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid'] {'action': ['delete'], 'not_if': 'ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1'}
2016-03-31 16:03:05,373 - Deleting File['/var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid']
2016-03-31 16:03:05,373 - Execute['ulimit -c unlimited; export HADOOP_LIBEXEC_DIR=/usr/hdp/current/hadoop-client/libexec && /usr/hdp/current/hadoop-yarn-client/sbin/yarn-daemon.sh --config /etc/hadoop/conf start nodemanager'] {'not_if': 'ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1', 'user': 'yarn'}
2016-03-31 16:03:06,427 - Execute['ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1'] {'initial_wait': 5, 'not_if': 'ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1', 'user': 'yarn'}
2016-03-31 16:03:11,470 - Error while executing command 'start':
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 123, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/YARN/package/scripts/nodemanager.py", line 42, in start
    action='start'
  File "/data/log/ambari-agent-1.0/cache/stacks/HDP/2.0.6/services/YARN/package/scripts/service.py", line 59, in service
    initial_wait=5
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 149, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 115, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 241, in action_run
    raise ex
Fail: Execution of 'ls /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid >/dev/null 2>&1 && ps `cat /var/run/hadoop-yarn/yarn/yarn-yarn-nodemanager.pid` >/dev/null 2>&1' returned 1. /etc/profile: line 81: ulimit: open files: cannot modify limit: Operation not permitted
2 REPLIES 2

Re: nodemanager start failed: ulimit: open files: cannot modify limit: Operation not permitted

@dash wei,

Could you try the below one and retry ?

Set hard limit as "ulimited" in /etc/limits.conf

Re: nodemanager start failed: ulimit: open files: cannot modify limit: Operation not permitted

New Contributor

thanks for your solution,jramakrishnan

I've modifed ulimit's nofile setting to 65536 of user:yarn successfully(CentOS Linux release 6.2 (Final).

But ulimit setting restored to 32768 after I restarted nodemanger by ambari UI. Nodemanger service start failed again.

Don't have an account?
Coming from Hortonworks? Activate your account here