Created 08-01-2017 02:26 PM
Hello,
I recently upgraded ambari to 2.4 did all the post upgrade stuff and now when im restarting services oozie fails and wont let me restart the other services. i thought it was a kerberos issue so i went to disable that since its not really needed and when it goes to shut down oozie it again fails. no idea why. below is the error out put. it seems to be looking for a file or configuration but i have no idea where it is located? any help would be appreciated.
Thanks in advance.
Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_client.py", line 76, in <module> OozieClient().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute method(env) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 680, in restart self.install(env) File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_client.py", line 38, in install self.configure(env) File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_client.py", line 45, in configure oozie(is_server=False) File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk return fn(*args, **kwargs) File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py", line 143, in oozie content=Template("oozie.conf.j2") File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 123, in action_create content = self._get_content() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 160, in _get_content return content() File "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 51, in __call__ return self.get_content() File "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 142, in get_content rendered = self.template.render(self.context) File "/usr/lib/python2.6/site-packages/ambari_jinja2/environment.py", line 891, in render return self.environment.handle_exception(exc_info, True) File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/templates/oozie.conf.j2", line 35, in top-level template code {{oozie_user}} - nproc {{oozie_user_nproc_limit}} File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__ raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!") resource_management.core.exceptions.Fail: Configuration parameter 'oozie_user_nofile_limit' was not found in configurations dictionary!
Created on 08-01-2017 05:51 PM - edited 08-17-2019 09:07 PM
Looks like you are missing below parameter in oozie environment.
Please add it and restart required services via Ambari.
Created 08-01-2017 03:41 PM
Created 08-01-2017 03:53 PM
@Geoffrey Shelton Okot
in the oozie.conf file i have
oozie - nofile 32000
oozie - nproc 1600
i am using ambari to configure everything though and i dont see anything about the oozie file limit in the config section at all.
Created 08-01-2017 04:36 PM
Created 08-01-2017 05:33 PM
ok i changed the limits and restarted and got the same error.
i then took a look at the server and gave that a restart. got an error about a missing directory /var/tmp/oozie so i made that directory and cleared that error.
restarted the server
and got the same dang error for the server now:
File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py", line 215, in <module> OozieServer().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py", line 88, in start self.configure(env) File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py", line 82, in configure oozie(is_server=True) File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk return fn(*args, **kwargs) File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py", line 143, in oozie content=Template("oozie.conf.j2") File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 123, in action_create content = self._get_content() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 160, in _get_content return content() File "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 51, in __call__ return self.get_content() File "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 142, in get_content rendered = self.template.render(self.context) File "/usr/lib/python2.6/site-packages/ambari_jinja2/environment.py", line 891, in render return self.environment.handle_exception(exc_info, True) File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/templates/oozie.conf.j2", line 35, in top-level template code {{oozie_user}} - nproc {{oozie_user_nproc_limit}} File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__ raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!") resource_management.core.exceptions.Fail: Configuration parameter 'oozie_user_nofile_limit' was not found in configurations dictionary!
here is the stdout for the server if that might help?
2017-08-01 13:15:51,918 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.3.0.0-2557 2017-08-01 13:15:51,918 - Checking if need to create versioned conf dir /etc/hadoop/2.3.0.0-2557/0 2017-08-01 13:15:51,919 - call[('ambari-python-wrap', '/usr/bin/conf-select', 'create-conf-dir', '--package', 'hadoop', '--stack-version', '2.3.0.0-2557', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1} 2017-08-01 13:15:51,952 - call returned (1, '/etc/hadoop/2.3.0.0-2557/0 exist already', '') 2017-08-01 13:15:51,953 - checked_call[('ambari-python-wrap', '/usr/bin/conf-select', 'set-conf-dir', '--package', 'hadoop', '--stack-version', '2.3.0.0-2557', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False} 2017-08-01 13:15:51,987 - checked_call returned (0, '') 2017-08-01 13:15:51,989 - Ensuring that hadoop has the correct symlink structure 2017-08-01 13:15:51,989 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2017-08-01 13:15:52,141 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.3.0.0-2557 2017-08-01 13:15:52,142 - Checking if need to create versioned conf dir /etc/hadoop/2.3.0.0-2557/0 2017-08-01 13:15:52,142 - call[('ambari-python-wrap', '/usr/bin/conf-select', 'create-conf-dir', '--package', 'hadoop', '--stack-version', '2.3.0.0-2557', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1} 2017-08-01 13:15:52,174 - call returned (1, '/etc/hadoop/2.3.0.0-2557/0 exist already', '') 2017-08-01 13:15:52,174 - checked_call[('ambari-python-wrap', '/usr/bin/conf-select', 'set-conf-dir', '--package', 'hadoop', '--stack-version', '2.3.0.0-2557', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False} 2017-08-01 13:15:52,209 - checked_call returned (0, '') 2017-08-01 13:15:52,210 - Ensuring that hadoop has the correct symlink structure 2017-08-01 13:15:52,211 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2017-08-01 13:15:52,213 - Group['spark'] {} 2017-08-01 13:15:52,216 - Group['ranger'] {} 2017-08-01 13:15:52,216 - Group['hadoop'] {} 2017-08-01 13:15:52,217 - Group['users'] {} 2017-08-01 13:15:52,217 - Group['knox'] {} 2017-08-01 13:15:52,218 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2017-08-01 13:15:52,219 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2017-08-01 13:15:52,220 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2017-08-01 13:15:52,221 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']} 2017-08-01 13:15:52,222 - User['atlas'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2017-08-01 13:15:52,222 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2017-08-01 13:15:52,223 - User['falcon'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']} 2017-08-01 13:15:52,225 - User['ranger'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['ranger']} 2017-08-01 13:15:52,226 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']} 2017-08-01 13:15:52,228 - User['mahout'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2017-08-01 13:15:52,229 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2017-08-01 13:15:52,231 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']} 2017-08-01 13:15:52,233 - User['flume'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2017-08-01 13:15:52,234 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2017-08-01 13:15:52,236 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2017-08-01 13:15:52,238 - User['sqoop'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2017-08-01 13:15:52,239 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2017-08-01 13:15:52,241 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2017-08-01 13:15:52,242 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2017-08-01 13:15:52,244 - User['knox'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2017-08-01 13:15:52,246 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']} 2017-08-01 13:15:52,247 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2017-08-01 13:15:52,251 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'} 2017-08-01 13:15:52,261 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if 2017-08-01 13:15:52,262 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'} 2017-08-01 13:15:52,264 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2017-08-01 13:15:52,267 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'} 2017-08-01 13:15:52,276 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if 2017-08-01 13:15:52,277 - Group['hdfs'] {} 2017-08-01 13:15:52,277 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'hdfs']} 2017-08-01 13:15:52,279 - FS Type: 2017-08-01 13:15:52,279 - Directory['/etc/hadoop'] {'mode': 0755} 2017-08-01 13:15:52,308 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'root', 'group': 'hadoop'} 2017-08-01 13:15:52,309 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777} 2017-08-01 13:15:52,324 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'} 2017-08-01 13:15:52,334 - Skipping Execute[('setenforce', '0')] due to not_if 2017-08-01 13:15:52,334 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'} 2017-08-01 13:15:52,339 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'} 2017-08-01 13:15:52,340 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'} 2017-08-01 13:15:52,350 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'root'} 2017-08-01 13:15:52,352 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'root'} 2017-08-01 13:15:52,353 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': ..., 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644} 2017-08-01 13:15:52,371 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs', 'group': 'hadoop'} 2017-08-01 13:15:52,372 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755} 2017-08-01 13:15:52,373 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'} 2017-08-01 13:15:52,379 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'} 2017-08-01 13:15:52,386 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755} 2017-08-01 13:15:52,643 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.3.0.0-2557 2017-08-01 13:15:52,643 - Checking if need to create versioned conf dir /etc/hadoop/2.3.0.0-2557/0 2017-08-01 13:15:52,644 - call[('ambari-python-wrap', '/usr/bin/conf-select', 'create-conf-dir', '--package', 'hadoop', '--stack-version', '2.3.0.0-2557', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False, 'stderr': -1} 2017-08-01 13:15:52,671 - call returned (1, '/etc/hadoop/2.3.0.0-2557/0 exist already', '') 2017-08-01 13:15:52,672 - checked_call[('ambari-python-wrap', '/usr/bin/conf-select', 'set-conf-dir', '--package', 'hadoop', '--stack-version', '2.3.0.0-2557', '--conf-version', '0')] {'logoutput': False, 'sudo': True, 'quiet': False} 2017-08-01 13:15:52,701 - checked_call returned (0, '') 2017-08-01 13:15:52,702 - Ensuring that hadoop has the correct symlink structure 2017-08-01 13:15:52,702 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2017-08-01 13:15:52,712 - checked_call['rpm -q --queryformat '%{version}-%{release}' hdp-select | sed -e 's/\.el[0-9]//g''] {'stderr': -1} 2017-08-01 13:15:52,762 - checked_call returned (0, '2.5.6.0-40', '') 2017-08-01 13:15:52,768 - HdfsResource['/user/oozie'] {'security_enabled': True, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': '/etc/security/keytabs/hdfs.headless.keytab', 'dfs_type': '', 'default_fs': 'hdfs://amaya.dge.local:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': 'hdfs@DGE.LOCAL', 'user': 'hdfs', 'owner': 'oozie', 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'action': ['create_on_execute'], 'immutable_paths': [u'/apps/hive/warehouse', u'/apps/falcon', u'/mr-history/done', u'/app-logs', u'/tmp'], 'mode': 0775} 2017-08-01 13:15:52,770 - Execute['/usr/bin/kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs@DGE.LOCAL'] {'user': 'hdfs'} 2017-08-01 13:15:52,835 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET --negotiate -u : '"'"'http://amaya.dge.local:50070/webhdfs/v1/user/oozie?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpUr0RTf 2>/tmp/tmpp13S1c''] {'logoutput': None, 'quiet': False} 2017-08-01 13:15:52,887 - call returned (0, '') 2017-08-01 13:15:52,888 - HdfsResource[None] {'security_enabled': True, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': '/etc/security/keytabs/hdfs.headless.keytab', 'dfs_type': '', 'default_fs': 'hdfs://amaya.dge.local:8020', 'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': 'hdfs@DGE.LOCAL', 'user': 'hdfs', 'action': ['execute'], 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'immutable_paths': [u'/apps/hive/warehouse', u'/apps/falcon', u'/mr-history/done', u'/app-logs', u'/tmp']} 2017-08-01 13:15:52,889 - Directory['/usr/hdp/current/oozie-server/conf'] {'owner': 'oozie', 'create_parents': True, 'group': 'hadoop'} 2017-08-01 13:15:52,889 - XmlConfig['oozie-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/oozie-server/conf', 'mode': 0664, 'configuration_attributes': {}, 'owner': 'oozie', 'configurations': ...} 2017-08-01 13:15:52,906 - Generating config: /usr/hdp/current/oozie-server/conf/oozie-site.xml 2017-08-01 13:15:52,906 - File['/usr/hdp/current/oozie-server/conf/oozie-site.xml'] {'owner': 'oozie', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': 0664, 'encoding': 'UTF-8'} 2017-08-01 13:15:52,947 - File['/usr/hdp/current/oozie-server/conf/oozie-env.sh'] {'content': InlineTemplate(...), 'owner': 'oozie', 'group': 'hadoop'} 2017-08-01 13:15:52,948 - Directory['/etc/security/limits.d'] {'owner': 'root', 'create_parents': True, 'group': 'root'} 2017-08-01 13:15:52,952 - File['/etc/security/limits.d/oozie.conf'] {'content': Template('oozie.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644} Command failed after 1 tries
could it be a kerberos issue? ive been thinking of removing it, its not really needed since the cluster is in a secure network. however i couldnt remove it because of oozie not wanting to shut down. however since i did the server restart it stopped working so i guess i can proceed with that? lol
thanks so much for your help its really appreciated.
Created on 08-01-2017 05:51 PM - edited 08-17-2019 09:07 PM
Looks like you are missing below parameter in oozie environment.
Please add it and restart required services via Ambari.
Created on 08-01-2017 05:51 PM - edited 08-17-2019 09:07 PM
Looks like you are missing below parameter in oozie environment.
Please add it and restart required services via Ambari.
Created on 08-01-2017 06:07 PM - edited 08-17-2019 09:07 PM
i dont have that variable in ambari ?
here is what my advanced oozie-env looks like
Created on 08-01-2017 06:18 PM - edited 08-17-2019 09:07 PM
You can add it via configs.sh/configs.py script
Here is my configuration for your reference.
Created 08-01-2017 07:29 PM