question Re: Enable Kerberos namenode start failed in Support Questions

Enable Kerberos namenode start failed

amazon20062007 — Wed, 28 Dec 2016 10:54:47 GMT

16/12/28 10:45:26 WARN retry.RetryInvocationHandler: Exception while invoking ClientNamenodeProtocolTranslatorPB.setSafeMode over null. Not retrying because try once and fail.

java.io.IOException: Failed on local exception: java.io.IOException: Couldn't setup connection for hdfs-hdpcluster@EXAMPLE.COM to bigdata013.example.com/<ip-address>:8020; Host Details : local host is: "bigdata013.example.com/<ip-address>"; destination host is: "bigdata013.example.com":8020; 
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:782)
	at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1556)
	at org.apache.hadoop.ipc.Client.call(Client.java:1496)
	at org.apache.hadoop.ipc.Client.call(Client.java:1396)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
	at com.sun.proxy.$Proxy10.setSafeMode(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.setSafeMode(ClientNamenodeProtocolTranslatorPB.java:711)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176)
	at com.sun.proxy.$Proxy11.setSafeMode(Unknown Source)
	at org.apache.hadoop.hdfs.DFSClient.setSafeMode(DFSClient.java:2657)
	at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:1340)
	at org.apache.hadoop.hdfs.DistributedFileSystem.setSafeMode(DistributedFileSystem.java:1324)
	at org.apache.hadoop.hdfs.tools.DFSAdmin.setSafeMode(DFSAdmin.java:611)
	at org.apache.hadoop.hdfs.tools.DFSAdmin.run(DFSAdmin.java:1916)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
	at org.apache.hadoop.hdfs.tools.DFSAdmin.main(DFSAdmin.java:2107)
Caused by: java.io.IOException: Couldn't setup connection for hdfs-hdpcluster@EXAMPLE.COM to bigdata013.example.com/<ip-address>:8020
	at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:712)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
	at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:683)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:770)
	at org.apache.hadoop.ipc.Client$Connection.access$3200(Client.java:397)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1618)
	at org.apache.hadoop.ipc.Client.call(Client.java:1449)
	... 20 more
Caused by: org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed
	at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:375)
	at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:595)
	at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:397)
	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:762)
	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:758)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:757)
	... 23 more
safemode: Failed on local exception: java.io.IOException: Couldn't setup connection for hdfs-hdpcluster@EXAMPLE.COM to bigdata013.example.com/<ip-address>:8020; Host Details : local host is: "bigdata013.example.com/<ip-address>"; destination host is: "bigdata013.example.com":8020; 
16/12/28 10:45:40 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
16/12/28 10:45:43 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
16/12/28 10:45:44 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
16/12/28 10:45:48 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.

Re: Enable Kerberos namenode start failed

jsensharma — Wed, 28 Dec 2016 16:51:29 GMT

@Zhao Chaofeng

As this issue is basically related to "GSS initiate failed", Hence can you please check if you see a valid ticket and if you are able to do a "kinit" manually?

Also are you using Sun JDK? If yes then you will have to install the JCE policies for encryption.

Please check the below link which says "Before enabling Kerberos in the cluster, you must deploy the Java Cryptography Extension (JCE) security policy files on the Ambari Server and on all hosts in the cluster."

: https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-security/content/installing_the_jce.html

Re: Enable Kerberos namenode start failed

amazon20062007 — Wed, 28 Dec 2016 16:58:31 GMT

Yes, I have installed JCE manually. And I execute "kinit" command to test ticket, the result is OK.

I have a question that whether KDC and ambari-server are in the same host, is it OK?

Re: Enable Kerberos namenode start failed

jsensharma — Wed, 28 Dec 2016 17:02:25 GMT

Yes, It is fine. KDC and Ambari can be co located on the same host. Or they can be remotely located as well.

I have a setup where i am running KDC and Ambari on the same host without any issue so far.

Re: Enable Kerberos namenode start failed

jsensharma — Wed, 28 Dec 2016 17:04:47 GMT

@Zhao Chaofeng

Since how long you are facing this issue? I mean is there a recent change happened? Or any recent upgrade?

What is your HDP and Ambari Version?

Only the NameNode is failing with the mentioned "GSS initiate failed", or few other components are also falling with the same issue.

I am sure that the Hostname is correct (example: hostname -f). Still it is worth to check.

- Is this kind of issue happening on the same host "bigdata013.example.com/<ip-address>:8020" ? Is this the only host (and the components hosted in this host) are giving the "GSS initiate failed" ? Or other hosts of your cluster are also having this issue? - Worth to check the hostname & KDC Connectivity.

Re: Enable Kerberos namenode start failed

amazon20062007 — Wed, 28 Dec 2016 17:17:03 GMT

No, I install HDP and ambari a minute ago. After installed, I "Enable Kerberos" and I face this issue.

HDP version: HDP-2.5.0.0

ambari version: Version 2.4.1.0

Of course, all service countered this issue.

I see your reply answer in my another question. After I install JCE, I encouter 'App Timeline Server start failed'.

The log is:

File"/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py", line 155,in<module>
ApplicationTimelineServer().execute()
File"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 280,in execute
method(env)
File"/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py", line 44,in start
self.configure(env)# FOR SECURITY
File"/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py", line 55,in configure
yarn(name='apptimelineserver')
File"/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89,in thunk
return fn(*args,**kwargs)
File"/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py", line 337,in yarn
mode=0755
File"/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155,in __init__
self.env.run()
File"/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160,in run
self.run_action(resource, action)
File"/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124,in run_action
provider_action()
File"/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 459,in action_create_on_execute
self.action_delayed("create")
File"/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 456,in action_delayed
self.get_hdfs_resource_executor().action_delayed(action_name,self)
File"/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 247,in action_delayed
self._assert_valid()
File"/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 231,in _assert_valid
self.target_status =self._get_file_status(target)
File"/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 292,in _get_file_status
list_status =self.util.run_command(target,'GETFILESTATUS', method='GET', ignore_status_codes=['404'], assertable_result=False)
File"/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 192,in run_command
raiseFail(err_msg)
resource_management.core.exceptions.Fail:Execution of 'curl -sS -L -w '%{http_code}' -X GET --negotiate -u : 'http://bigdata013.example.com:50070/webhdfs/v1/ats/done?op=GETFILESTATUS&user.name=hdfs'' returned status_code=403.
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error403GSSException:Failure unspecified at GSS-API level (Mechanism level:Encryption type AES256 CTS mode with HMAC SHA1-96isnot supported/enabled)</title>
</head>
<body><h2>HTTP ERROR 403</h2>
Problem accessing /webhdfs/v1/ats/done.Reason:
<pre>GSSException:Failure unspecified at GSS-API level (Mechanism level:Encryption type AES256 CTS mode with HMAC SHA1-96isnot supported/enabled)</pre><hr />PoweredbyJetty://
</body>
</html>

Re: Enable Kerberos namenode start failed

jsensharma — Wed, 28 Dec 2016 17:20:11 GMT

@Zhao Chaofeng

- Please check if the "Reverse lookup" is correct or not on that host?

- Also it will be best if you can share the output of to see the kerberos related JAVA options used.

 ps -ef | grep AmbariServer
 ps -ef | grep NameNode

- Just to verify the correct JAVA Path which has the JCE.

Re: Enable Kerberos namenode start failed

jsensharma — Wed, 28 Dec 2016 17:24:09 GMT

@Zhao Chaofeng

Also can you please share the output of the following command "klist -e -k /etc/security/keytabs/hdfs.headless.keytab"

to see the encryption types used by the kerberos tickets?

# klist -e -k /etc/security/keytabs/hdfs.headless.keytab
Keytab name: FILE:/etc/security/keytabs/hdfs.headless.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   4 hdfs-JoyCluster@EXAMPLE.COM (des3-cbc-sha1) 
   4 hdfs-JoyCluster@EXAMPLE.COM (arcfour-hmac) 
   4 hdfs-JoyCluster@EXAMPLE.COM (aes128-cts-hmac-sha1-96) 
   4 hdfs-JoyCluster@EXAMPLE.COM (des-cbc-md5) 
   4 hdfs-JoyCluster@EXAMPLE.COM (aes256-cts-hmac-sha1-96)

Re: Enable Kerberos namenode start failed

amazon20062007 — Wed, 28 Dec 2016 18:09:22 GMT

NameNode is in safe mode, and it can not up.

Re: Enable Kerberos namenode start failed

amazon20062007 — Wed, 28 Dec 2016 18:10:45 GMT

Keytab name: FILE:/etc/security/keytabs/hdfs.headless.keytab

KVNO Principal

---- --------------------------------------------------------------------------

1 hdfs-hdpcluster@EXAMPLE.COM (des3-cbc-sha1)

1 hdfs-hdpcluster@EXAMPLE.COM (aes256-cts-hmac-sha1-96)

1 hdfs-hdpcluster@EXAMPLE.COM (arcfour-hmac)

1 hdfs-hdpcluster@EXAMPLE.COM (aes128-cts-hmac-sha1-96) 1 hdfs-hdpcluster@EXAMPLE.COM (des-cbc-md5)

Re: Enable Kerberos namenode start failed

amazon20062007 — Wed, 28 Dec 2016 18:19:54 GMT

After I reinstall kerberos, it looks like ok and I start NameNode and install other service also good. But I can not find why it failed previous.

Thank you for your help.

Re: Enable Kerberos namenode start failed

sumit_nigam — Mon, 23 Jan 2017 17:26:20 GMT

@Zhao Chaofeng - I am having exact problem with Hbase (1.1.2) - GSS Initiate failed even with a valid kerberos service ticket. Can you please let me know what version of kerberos libraries did you re-install? I am using Kerberos 5 version 1.10.3

Thanks! Sumit

Re: Enable Kerberos namenode start failed

amazon20062007 — Mon, 23 Jan 2017 17:53:05 GMT

Kerberos 1.15. I install it with repo of CentOS 7.

Re: Enable Kerberos namenode start failed

frederic_wou — Thu, 02 Nov 2017 16:52:25 GMT

Check your "/etckrb5.conf"

("default_tgs_enctypes" ; "default_tkt_enctypes" ; "permitted_enctypes" must contain encryption types you use.

Renewing your ticket can help ...

here for example, launch this command :

kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs-hdp_dev@EXAMPLE.COM

Re: Enable Kerberos namenode start failed

Shelton — Thu, 02 Nov 2017 19:29:32 GMT

@Zhao Chaofeng

Is this thread still open? i.e hasn't this problem been resolved?

Please revert