Member since
06-29-2017
14
Posts
0
Kudos Received
0
Solutions
08-04-2017
02:59 PM
@Geoffrey Shelton Okot so i dug into the ranger-admin logs and i see this error being repeated. 2017-08-03 14:07:29,908 [<server name removed>l-startStop-1] FATAL org.apache.ranger.plugin.store.EmbeddedServiceDefsUtil (EmbeddedServiceDefsUtil.java:181) - EmbeddedSe$
javax.persistence.PersistenceException:
Exception [EclipseLink-4002] (Eclipse Persistence Services -
2.5.2.v20131113-a7346c6): org.eclipse.persistence.exceptions.Datab$
Internal Exception: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Table 'ranger.x_service_def' doesn't exist
Error Code: 1146
it seems a table is missing some where thats throwing an error code? would it just be easier to remove the ranger service and reinstall it? ranger admin is running on my ambari-server machine as well. dont know if that might cause issues.
... View more
08-04-2017
02:16 PM
@Geoffrey Shelton Okot yeah it looks all good. i turned off iptables just to make sure it wasnt a firewall issue. the ranger admin is on the same server as ambari-server as well and ambari is working fine. the URL matches with my hosts files as well.
... View more
08-03-2017
07:17 PM
@Geoffrey Shelton Okot yup it is there http://<servers fqdn>:6080 i have ldap checked should that be set to none to auth?
... View more
08-03-2017
06:51 PM
ok so i got ranger admin and user sync running however i have two amber alerts about the ranger WEB UI giving a 404 and i cant access it. i ranger the service check on the ranger service and this is the error message i get. stderr: /var/lib/ambari-agent/data/errors-4899.txt
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/service_check.py", line 49, in <module>
RangerServiceCheck().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/service_check.py", line 34, in service_check
self.check_ranger_admin_service(params.ranger_external_url, params.upgrade_marker_file)
File "/var/lib/ambari-agent/cache/common-services/RANGER/0.4.0/package/scripts/service_check.py", line 43, in check_ranger_admin_service
logoutput=True)
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 273, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 71, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 93, in checked_call
tries=tries, try_sleep=try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 141, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 294, in _call
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'curl -s -o /dev/null -w'%{http_code}' --negotiate -u: -k http://<server name removed>:6080/login.jsp | grep 200' returned 1.
so it seems for some reason it can not connect to the Ranger web UI any thoughts on why that is?
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Ranger
08-02-2017
07:54 PM
ahhhhh yes! ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar ran that command and it worked. guess when it was updated it broke that connection? either way its working now! thanks so much!
... View more
08-02-2017
07:31 PM
recently upgraded to ambari 2.4 got though a previous oozie issue and now when im restarting hive i get this error for hiver-server2: stderr: /var/lib/ambari-agent/data/errors-4864.txt
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py", line 211, in <module>
HiveServer().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute
method(env)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 720, in restart
self.start(env, upgrade_type=upgrade_type)
File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_server.py", line 88, in start
hive_service('hiveserver2', action = 'start', upgrade_type=upgrade_type)
File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
return fn(*args, **kwargs)
File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py", line 106, in hive_service
validate_connection(params.hive_jdbc_target, params.hive_lib)
File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_service.py", line 162, in validate_connection
path='/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin', tries=5, try_sleep=10)
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 273, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 71, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 93, in checked_call
tries=tries, try_sleep=try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 141, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 294, in _call
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of '/usr/jdk64/jdk1.8.0_40/bin/java -cp /usr/lib/ambari-agent/DBConnectionVerification.jar:/usr/hdp/current/hive-server2/lib/mysql-connector-java.jar org.apache.ambari.server.DBConnectionVerification 'jdbc:mysql://<server name removed>/hive?createDatabaseIfNotExist=true' hive [PROTECTED] com.mysql.jdbc.Driver' returned 1. ERROR: Unable to connect to the DB. Please check DB connection properties.
java.lang.ClassNotFoundException: com.mysql.jdbc.Driver i inherited this cluster so i have no idea what DB it was connecting and i cant for the life of me find anything about it in the documentation i was left. knox and ranger are not enabled but it seems kerberos is but nothing has changed with that.
... View more
Labels:
- Labels:
-
Apache Hive
08-02-2017
02:22 PM
@Kuldeep Kulkarni /var/lib/ambari-server/resources/scripts/configs.sh
-u <ambari-admin-username> -p <ambari-admin-password> set <ambari-server-hostname> <cluster-name> ozie-env oozie_user_nofile_limit 32000
and oozie_user_nproc_limit 16000 had to remove the word advanced its just oozie-env 🙂 but its all working now! thank you soooooo much!!! and thank you @Geoffrey Shelton Okot for the help as well!!! you guys are awesome im very grateful for the help.
... View more
08-01-2017
07:29 PM
@Kuldeep Kulkarni do you happen to have something on how to do that? never done something like before.
... View more
08-01-2017
06:07 PM
@Kuldeep Kulkarni i dont have that variable in ambari ? here is what my advanced oozie-env looks like
... View more
08-01-2017
05:33 PM
@Geoffrey Shelton Okot ok i changed the limits and restarted and got the same error. i then took a look at the server and gave that a restart. got an error about a missing directory /var/tmp/oozie so i made that directory and cleared that error. restarted the server and got the same dang error for the server now: File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py", line 215, in <module>
OozieServer().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py", line 88, in start
self.configure(env)
File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_server.py", line 82, in configure
oozie(is_server=True)
File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
return fn(*args, **kwargs)
File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py", line 143, in oozie
content=Template("oozie.conf.j2")
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 123, in action_create
content = self._get_content()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 160, in _get_content
return content()
File "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 51, in __call__
return self.get_content()
File "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 142, in get_content
rendered = self.template.render(self.context)
File "/usr/lib/python2.6/site-packages/ambari_jinja2/environment.py", line 891, in render
return self.environment.handle_exception(exc_info, True)
File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/templates/oozie.conf.j2", line 35, in top-level template code
{{oozie_user}} - nproc {{oozie_user_nproc_limit}}
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__
raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 'oozie_user_nofile_limit' was not found in configurations dictionary!
here is the stdout for the server if that might help? 2017-08-01 13:15:51,918 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.3.0.0-2557
2017-08-01 13:15:51,918 - Checking if need to create versioned conf dir /etc/hadoop/2.3.0.0-2557/0
2017-08-01
13:15:51,919 - call[('ambari-python-wrap', '/usr/bin/conf-select',
'create-conf-dir', '--package', 'hadoop', '--stack-version',
'2.3.0.0-2557', '--conf-version', '0')] {'logoutput': False, 'sudo':
True, 'quiet': False, 'stderr': -1}
2017-08-01 13:15:51,952 - call returned (1, '/etc/hadoop/2.3.0.0-2557/0 exist already', '')
2017-08-01 13:15:51,953 - checked_call[('ambari-python-wrap',
'/usr/bin/conf-select', 'set-conf-dir', '--package', 'hadoop',
'--stack-version', '2.3.0.0-2557', '--conf-version', '0')] {'logoutput':
False, 'sudo': True, 'quiet': False}
2017-08-01 13:15:51,987 - checked_call returned (0, '')
2017-08-01 13:15:51,989 - Ensuring that hadoop has the correct symlink structure
2017-08-01 13:15:51,989 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-08-01 13:15:52,141 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.3.0.0-2557
2017-08-01 13:15:52,142 - Checking if need to create versioned conf dir /etc/hadoop/2.3.0.0-2557/0
2017-08-01
13:15:52,142 - call[('ambari-python-wrap', '/usr/bin/conf-select',
'create-conf-dir', '--package', 'hadoop', '--stack-version',
'2.3.0.0-2557', '--conf-version', '0')] {'logoutput': False, 'sudo':
True, 'quiet': False, 'stderr': -1}
2017-08-01 13:15:52,174 - call returned (1, '/etc/hadoop/2.3.0.0-2557/0 exist already', '')
2017-08-01 13:15:52,174 - checked_call[('ambari-python-wrap',
'/usr/bin/conf-select', 'set-conf-dir', '--package', 'hadoop',
'--stack-version', '2.3.0.0-2557', '--conf-version', '0')] {'logoutput':
False, 'sudo': True, 'quiet': False}
2017-08-01 13:15:52,209 - checked_call returned (0, '')
2017-08-01 13:15:52,210 - Ensuring that hadoop has the correct symlink structure
2017-08-01 13:15:52,211 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-08-01 13:15:52,213 - Group['spark'] {}
2017-08-01 13:15:52,216 - Group['ranger'] {}
2017-08-01 13:15:52,216 - Group['hadoop'] {}
2017-08-01 13:15:52,217 - Group['users'] {}
2017-08-01 13:15:52,217 - Group['knox'] {}
2017-08-01 13:15:52,218 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-08-01 13:15:52,219 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-08-01 13:15:52,220 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-08-01 13:15:52,221 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']}
2017-08-01 13:15:52,222 - User['atlas'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-08-01 13:15:52,222 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-08-01 13:15:52,223 - User['falcon'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']}
2017-08-01 13:15:52,225 - User['ranger'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['ranger']}
2017-08-01 13:15:52,226 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']}
2017-08-01 13:15:52,228 - User['mahout'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-08-01 13:15:52,229 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-08-01 13:15:52,231 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['users']}
2017-08-01 13:15:52,233 - User['flume'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-08-01 13:15:52,234 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-08-01 13:15:52,236 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-08-01 13:15:52,238 - User['sqoop'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-08-01 13:15:52,239 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-08-01 13:15:52,241 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-08-01 13:15:52,242 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-08-01 13:15:52,244 - User['knox'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-08-01 13:15:52,246 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
2017-08-01 13:15:52,247 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-08-01 13:15:52,251 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2017-08-01 13:15:52,261 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2017-08-01
13:15:52,262 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase',
'create_parents': True, 'mode': 0775, 'cd_access': 'a'}
2017-08-01 13:15:52,264 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-08-01 13:15:52,267 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2017-08-01 13:15:52,276 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if
2017-08-01 13:15:52,277 - Group['hdfs'] {}
2017-08-01 13:15:52,277 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'hdfs']}
2017-08-01 13:15:52,279 - FS Type:
2017-08-01 13:15:52,279 - Directory['/etc/hadoop'] {'mode': 0755}
2017-08-01 13:15:52,308 - File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'root', 'group': 'hadoop'}
2017-08-01 13:15:52,309 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2017-08-01
13:15:52,324 - Execute[('setenforce', '0')] {'not_if': '(! which
getenforce ) || (which getenforce && getenforce | grep -q
Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2017-08-01 13:15:52,334 - Skipping Execute[('setenforce', '0')] due to not_if
2017-08-01
13:15:52,334 - Directory['/var/log/hadoop'] {'owner': 'root',
'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access':
'a'}
2017-08-01 13:15:52,339 - Directory['/var/run/hadoop']
{'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access':
'a'}
2017-08-01 13:15:52,340 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
2017-08-01 13:15:52,350 - File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'root'}
2017-08-01 13:15:52,352 - File['/usr/hdp/current/hadoop-client/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'root'}
2017-08-01 13:15:52,353 - File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] {'content': ..., 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2017-08-01 13:15:52,371 - File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties'] {'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs', 'group': 'hadoop'}
2017-08-01 13:15:52,372 - File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2017-08-01 13:15:52,373 - File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2017-08-01 13:15:52,379 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'}
2017-08-01 13:15:52,386 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2017-08-01 13:15:52,643 - The hadoop conf dir /usr/hdp/current/hadoop-client/conf exists, will call conf-select on it for version 2.3.0.0-2557
2017-08-01 13:15:52,643 - Checking if need to create versioned conf dir /etc/hadoop/2.3.0.0-2557/0
2017-08-01
13:15:52,644 - call[('ambari-python-wrap', '/usr/bin/conf-select',
'create-conf-dir', '--package', 'hadoop', '--stack-version',
'2.3.0.0-2557', '--conf-version', '0')] {'logoutput': False, 'sudo':
True, 'quiet': False, 'stderr': -1}
2017-08-01 13:15:52,671 - call returned (1, '/etc/hadoop/2.3.0.0-2557/0 exist already', '')
2017-08-01 13:15:52,672 - checked_call[('ambari-python-wrap',
'/usr/bin/conf-select', 'set-conf-dir', '--package', 'hadoop',
'--stack-version', '2.3.0.0-2557', '--conf-version', '0')] {'logoutput':
False, 'sudo': True, 'quiet': False}
2017-08-01 13:15:52,701 - checked_call returned (0, '')
2017-08-01 13:15:52,702 - Ensuring that hadoop has the correct symlink structure
2017-08-01 13:15:52,702 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-08-01
13:15:52,712 - checked_call['rpm -q --queryformat
'%{version}-%{release}' hdp-select | sed -e 's/\.el[0-9]//g'']
{'stderr': -1}
2017-08-01 13:15:52,762 - checked_call returned (0, '2.5.6.0-40', '')
2017-08-01 13:15:52,768 - HdfsResource['/user/oozie'] {'security_enabled': True, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': '/etc/security/keytabs/hdfs.headless.keytab',
'dfs_type': '', 'default_fs': 'hdfs://amaya.dge.local:8020',
'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore',
'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit',
'principal_name': 'hdfs@DGE.LOCAL', 'user': 'hdfs', 'owner': 'oozie',
'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type':
'directory', 'action': ['create_on_execute'], 'immutable_paths':
[u'/apps/hive/warehouse', u'/apps/falcon', u'/mr-history/done',
u'/app-logs', u'/tmp'], 'mode': 0775}
2017-08-01 13:15:52,770 - Execute['/usr/bin/kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs@DGE.LOCAL'] {'user': 'hdfs'}
2017-08-01
13:15:52,835 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl
-sS -L -w '"'"'%{http_code}'"'"' -X GET --negotiate -u : '"'"'http://amaya.dge.local:50070/webhdfs/v1/user/oozie?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpUr0RTf 2>/tmp/tmpp13S1c''] {'logoutput': None, 'quiet': False}
2017-08-01 13:15:52,887 - call returned (0, '')
2017-08-01 13:15:52,888 - HdfsResource[None] {'security_enabled': True, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': '/etc/security/keytabs/hdfs.headless.keytab',
'dfs_type': '', 'default_fs': 'hdfs://amaya.dge.local:8020',
'hdfs_resource_ignore_file': '/var/lib/ambari-agent/data/.hdfs_resource_ignore',
'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit',
'principal_name': 'hdfs@DGE.LOCAL', 'user': 'hdfs', 'action':
['execute'], 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'immutable_paths': [u'/apps/hive/warehouse', u'/apps/falcon', u'/mr-history/done', u'/app-logs', u'/tmp']}
2017-08-01 13:15:52,889 - Directory['/usr/hdp/current/oozie-server/conf'] {'owner': 'oozie', 'create_parents': True, 'group': 'hadoop'}
2017-08-01 13:15:52,889 - XmlConfig['oozie-site.xml'] {'group': 'hadoop', 'conf_dir': '/usr/hdp/current/oozie-server/conf', 'mode': 0664, 'configuration_attributes': {}, 'owner': 'oozie', 'configurations': ...}
2017-08-01 13:15:52,906 - Generating config: /usr/hdp/current/oozie-server/conf/oozie-site.xml
2017-08-01 13:15:52,906 - File['/usr/hdp/current/oozie-server/conf/oozie-site.xml']
{'owner': 'oozie', 'content': InlineTemplate(...), 'group': 'hadoop',
'mode': 0664, 'encoding': 'UTF-8'}
2017-08-01 13:15:52,947 - File['/usr/hdp/current/oozie-server/conf/oozie-env.sh'] {'content': InlineTemplate(...), 'owner': 'oozie', 'group': 'hadoop'}
2017-08-01 13:15:52,948 - Directory['/etc/security/limits.d'] {'owner': 'root', 'create_parents': True, 'group': 'root'}
2017-08-01 13:15:52,952 - File['/etc/security/limits.d/oozie.conf'] {'content': Template('oozie.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644}
Command failed after 1 tries
could it be a kerberos issue? ive been thinking of removing it, its not really needed since the cluster is in a secure network. however i couldnt remove it because of oozie not wanting to shut down. however since i did the server restart it stopped working so i guess i can proceed with that? lol thanks so much for your help its really appreciated.
... View more
08-01-2017
03:53 PM
@Geoffrey Shelton Okot in the oozie.conf file i have oozie - nofile 32000 oozie - nproc 1600 i am using ambari to configure everything though and i dont see anything about the oozie file limit in the config section at all.
... View more
08-01-2017
02:26 PM
Hello, I recently upgraded ambari to 2.4 did all the post upgrade stuff and now when im restarting services oozie fails and wont let me restart the other services. i thought it was a kerberos issue so i went to disable that since its not really needed and when it goes to shut down oozie it again fails. no idea why. below is the error out put. it seems to be looking for a file or configuration but i have no idea where it is located? any help would be appreciated. Thanks in advance. Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_client.py", line 76, in <module>
OozieClient().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute
method(env)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 680, in restart
self.install(env)
File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_client.py", line 38, in install
self.configure(env)
File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie_client.py", line 45, in configure
oozie(is_server=False)
File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
return fn(*args, **kwargs)
File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/scripts/oozie.py", line 143, in oozie
content=Template("oozie.conf.j2")
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 123, in action_create
content = self._get_content()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 160, in _get_content
return content()
File "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 51, in __call__
return self.get_content()
File "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 142, in get_content
rendered = self.template.render(self.context)
File "/usr/lib/python2.6/site-packages/ambari_jinja2/environment.py", line 891, in render
return self.environment.handle_exception(exc_info, True)
File "/var/lib/ambari-agent/cache/common-services/OOZIE/4.0.0.2.0/package/templates/oozie.conf.j2", line 35, in top-level template code
{{oozie_user}} - nproc {{oozie_user_nproc_limit}}
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__
raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 'oozie_user_nofile_limit' was not found in configurations dictionary!
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Oozie
06-29-2017
07:56 PM
im having the same issue. im going to give your solution a try. @Alan Featherston where do you change the hadoop_datanode_opts? im using ambari to control everything as well. thanks
... View more
06-29-2017
07:17 PM
this morning one of my datanodes was not sending heartbeats and the whole server seemed to have been locked up.
now when it came back i did the usual start up of things through ambari no issues everything came back green.
however 2 alerts popped up. one was the Datanode process and the other was the Datanode webui giving me an errno 1111 connect refused can not connect to 0.0.0.0:1022 connection refused in the past i could connect and see the datanode was working by putting <ip-address>:1022 in my webbrowser and it would take me to a screen showing its running. however its not allowing that anymore. ambari says its started but HDFS is showing a dead node and the webui isnt working.
i checked the logs and i cant for the life of me see any type of error. and there is no error in the start up process as well. so any help on troubleshooting and or fixing this issue would be great. thanks in advance.
... View more
Labels:
- Labels:
-
Apache Hadoop