Failed to Start NameNode 403 error

Hefei — Fri, 05 Jan 2018 00:46:15 GMT

I have installed Ambari 2.6.0.0 version with HDP 2.6.3 and added some of the services. But while enabling Kerberos I am facing one weird issue. The error log is:

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 361, in <module>
    NameNode().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 367, in execute
    method(env)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 970, in restart
    self.start(env, upgrade_type=upgrade_type)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/namenode.py", line 99, in start
    upgrade_suspended=params.upgrade_suspended, env=env)
  File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
    return fn(*args, **kwargs)
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py", line 234, in namenode
    create_hdfs_directories()
  File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/hdfs_namenode.py", line 301, in create_hdfs_directories
    mode=0777,
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 604, in action_create_on_execute
    self.action_delayed("create")
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 601, in action_delayed
    self.get_hdfs_resource_executor().action_delayed(action_name, self)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 328, in action_delayed
    self._assert_valid()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 287, in _assert_valid
    self.target_status = self._get_file_status(target)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 430, in _get_file_status
    list_status = self.util.run_command(target, 'GETFILESTATUS', method='GET', ignore_status_codes=['404'], assertable_result=False)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 177, in run_command
    return self._run_command(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 248, in _run_command
    raise WebHDFSCallException(err_msg, result_dict)
resource_management.libraries.providers.hdfs_resource.WebHDFSCallException: Execution of 'curl -sS -L -w '%{http_code}' -X GET --negotiate -u : 'http://a01-r03-i164-156-515w9ay.xxx.xxx:50070/webhdfs/v1/tmp?op=GETFILESTATUS'' returned status_code=403. 
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 403 org.apache.hadoop.security.authentication.client.AuthenticationException</title>
</head>
<body><h2>HTTP ERROR 403</h2>
<p>Problem accessing /webhdfs/v1/tmp. Reason:
<pre>    org.apache.hadoop.security.authentication.client.AuthenticationException</pre></p><hr /><i><small>Powered by Jetty://</small></i><br/>                                                
<br/>

Any suggestions will be appreciated. Thanks!

Re: Failed to Start NameNode 403 error

tletempt — Tue, 09 Jan 2018 11:35:04 GMT

Can you kinit successfully as the Namenode prinicpal? Does the Namenode Principal have the correct permissions? Is your krb5.conf in /etc or in a non-standard directory?

Re: Failed to Start NameNode 403 error

jsensharma — Tue, 09 Jan 2018 11:45:45 GMT

@Hefei Li

Please check the Kerberos Principals and also the Hostname. If the Hostname is in correct Case or not?

Sometimes this kind of error occurs if the "/etc/hosts" file has hostname in Mixed Case (LowerCase).

So please check if the NameNode host actually has the Hostname which is reported here "a01-r03-i164-156-515w9ay.xxx.xxx" ?

Also please check if your Keytabs are correctly reflecting the hostname/realm name in proper case.

# klist -kte /etc/security/keytabs/hdfs.headless.keytab
# hostname -f
# cat /etc/hosts

Please check the FQDN / Hostname which is mentioned in the above output matches the HDFS configuration files for the NameNode address.

Re: Failed to Start NameNode 403 error

Hefei — Fri, 12 Jan 2018 14:18:34 GMT

I have found the reason for the problem.

I changed the FQDN to lowercase, then it works ok.

Thank you for your reply

Re: Failed to Start NameNode 403 error

jsensharma — Fri, 12 Jan 2018 14:23:10 GMT

@Hefei Li

Wonderful!!! Good to see that the issue is resolved.

As the issue is resolved, hence it will be also great if you can mark this HCC thread as Answered by clicking on the "Accept" Button on the correct answer. That way other HCC users can quickly find the solution when they encounter the same issue.

question Re: Failed to Start NameNode 403 error in Archives of Support Questions (Read Only)

Failed to Start NameNode 403 error

Re: Failed to Start NameNode 403 error

Re: Failed to Start NameNode 403 error

Re: Failed to Start NameNode 403 error

Re: Failed to Start NameNode 403 error