SYMPTOMS: HDP running has recently been upgraded to Isilon 8.0.1.0 and kerberized. When trying to start services through Ambari, user receives the following stderr: Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py", line 289, in <module>
Resourcemanager().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py", line 124, in start
self.wait_for_dfs_directories_created(params.entity_groupfs_store_dir, params.entity_groupfs_active_dir)
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py", line 246, in wait_for_dfs_directories_created
self.wait_for_dfs_directory_created(dir_path, ignored_dfs_dirs)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/decorator.py", line 55, in wrapper
return function(*args, **kwargs)
File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py", line 268, in wait_for_dfs_directory_created
list_status = util.run_command(dir_path, 'GETFILESTATUS', method='GET', ignore_status_codes=['404'], assertable_result=False)
File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 192, in run_command
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X GET --negotiate -u : 'http://hwx.isilon.support:8082/webhdfs/v1/hwx/done/?op=GETFILESTATUS&user.name=hdfs'' returned status_code=401.
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>401 Authorization Required</title>
</head><body>
<h1>Authorization Required</h1>
<p>This server could not verify that you
are authorized to access the document
requested. Either you supplied the wrong
credentials (e.g., bad password), or your
browser doesn't understand how to supply
the credentials required.</p>
</body></html> Services are not able to authenticate. ROOT CAUSE: Isilon bug number: 84897 RESOLUTION: During Isilon 8.0.1.0 upgrade, permissions are incorrectly changed on /etc/krb5.conf to 660 (instead of 644). This causes WebHDFS to be unable to check the kerberos tickets. This issue is fixed in Isilon 8.0.1.1. The workaround was to change the permissions on /etc/krb5.conf to 644. Please reach out to Dell/EMC if you need assistance with the workaround.
... View more
