Support Questions

prijan_kurup · ‎12-10-2015

Hi

Getting the following error in the hdfs log files "hadoop-hdfs-datanode-hashmap.domain.com.log"

Error

javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
        at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
        at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:558)
        at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:373)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:727)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:723)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:722)
        at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:373)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1493)
        at org.apache.hadoop.ipc.Client.call(Client.java:1397)
        at org.apache.hadoop.ipc.Client.call(Client.java:1358)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
        at com.sun.proxy.$Proxy15.versionRequest(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.versionRequest(DatanodeProtocolClientSideTranslatorPB.java:272)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.retrieveNamespaceInfo(BPServiceActor.java:173)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:219)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:821)
        at java.lang.Thread.run(Thread.java:745)
Caused by: GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service crews)

Need help / pointers to fix the issue.

rlevas · ‎12-11-2015

On the host that is showing this error, can you klist the following keytab files:

/etc/security/keytabs/spnego.service.keytab
/etc/security/keytabs/dn.service.keytab

[root@c6501 ~]# klist -kte /etc/security/keytabs/spnego.service.keytab
Keytab name: FILE:/etc/security/keytabs/spnego.service.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   1 12/11/15 15:38:21 HTTP/c6501.ambari.apache.org@EXAMPLE.COM (des-cbc-md5)
   1 12/11/15 15:38:21 HTTP/c6501.ambari.apache.org@EXAMPLE.COM (aes256-cts-hmac-sha1-96)
   1 12/11/15 15:38:21 HTTP/c6501.ambari.apache.org@EXAMPLE.COM (des3-cbc-sha1)
   1 12/11/15 15:38:21 HTTP/c6501.ambari.apache.org@EXAMPLE.COM (arcfour-hmac)
   1 12/11/15 15:38:21 HTTP/c6501.ambari.apache.org@EXAMPLE.COM (aes128-cts-hmac-sha1-96)

[root@c6501 ~]# klist -kte /etc/security/keytabs/dn.service.keytab
Keytab name: FILE:/etc/security/keytabs/dn.service.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   1 12/11/15 15:38:21 dn/c6501.ambari.apache.org@EXAMPLE.COM (des-cbc-md5)
   1 12/11/15 15:38:21 dn/c6501.ambari.apache.org@EXAMPLE.COM (aes256-cts-hmac-sha1-96)
   1 12/11/15 15:38:21 dn/c6501.ambari.apache.org@EXAMPLE.COM (des3-cbc-sha1)
   1 12/11/15 15:38:21 dn/c6501.ambari.apache.org@EXAMPLE.COM (arcfour-hmac)
   1 12/11/15 15:38:21 dn/c6501.ambari.apache.org@EXAMPLE.COM (aes128-cts-hmac-sha1-96)

Notice that my examples have entries for aes256-cts-hmac-sha1-96, do yours? If not, then you will need to make sure that you have the Unlimited Key JCE policy jars (for your JVM) installed.

If these look good, then from the host you having the issue with, can you attempt to kinit using them. If you get no messages, all is good... else there is a disconnect between your keytab file and the password stored in the KDC.

Success case:

[root@c6501 ~]# kinit -kt /etc/security/keytabs/dn.service.keytab dn/c6501.ambari.apache.org@EXAMPLE.COM
[root@c6501 ~]#

Failure case:

[root@c6501 ~]# kinit -kt /etc/security/keytabs/dn.service.keytab dn/c6501.ambari.apache.org@EXAMPLE.COM
kinit: Password incorrect while getting initial credentials
[root@c6501 ~]#

If you get this failure, then try to regenerate the keytab files from the Ambari UI to see if that helps the issue.

View solution in original post

rlevas · ‎12-11-2015

Was Kerberos enabled using Ambari's automation, or was this done manually?

Cloudera Community

Support Questions

Services failing to restart after kerberizing cluster - HDP2.3