Community Articles

zstai · ‎12-29-2016

SYMPTOMS:

HDP running has recently been upgraded to Isilon 8.0.1.0 and kerberized.

When trying to start services through Ambari, user receives the following stderr:

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py", line 289, in <module>
    Resourcemanager().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py", line 124, in start
    self.wait_for_dfs_directories_created(params.entity_groupfs_store_dir, params.entity_groupfs_active_dir)
  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py", line 246, in wait_for_dfs_directories_created
    self.wait_for_dfs_directory_created(dir_path, ignored_dfs_dirs)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/decorator.py", line 55, in wrapper
    return function(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/resourcemanager.py", line 268, in wait_for_dfs_directory_created
    list_status = util.run_command(dir_path, 'GETFILESTATUS', method='GET', ignore_status_codes=['404'], assertable_result=False)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 192, in run_command
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X GET --negotiate -u : 'http://hwx.isilon.support:8082/webhdfs/v1/hwx/done/?op=GETFILESTATUS&user.name=hdfs'' returned status_code=401. 
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>401 Authorization Required</title>
</head><body>
<h1>Authorization Required</h1>
<p>This server could not verify that you
are authorized to access the document
requested.  Either you supplied the wrong
credentials (e.g., bad password), or your
browser doesn't understand how to supply
the credentials required.</p>
</body></html>

Services are not able to authenticate.

ROOT CAUSE: Isilon bug number: 84897

RESOLUTION: During Isilon 8.0.1.0 upgrade, permissions are incorrectly changed on /etc/krb5.conf to 660 (instead of 644). This causes WebHDFS to be unable to check the kerberos tickets. This issue is fixed in Isilon 8.0.1.1. The workaround was to change the permissions on /etc/krb5.conf to 644. Please reach out to Dell/EMC if you need assistance with the workaround.

Cloudera Community

Community Articles

After kerberizing HDP cluster with Isilon 8.0.1.0, several services will not start

Apache Hadoop