Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Services failing to restart after kerberizing cluster - HDP2.3

Solved Go to solution
Highlighted

Services failing to restart after kerberizing cluster - HDP2.3

Hi

Getting the following error in the hdfs log files "hadoop-hdfs-datanode-hashmap.domain.com.log"

Error

javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service creds)]
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
        at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
        at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:558)
        at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:373)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:727)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:723)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:722)
        at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:373)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1493)
        at org.apache.hadoop.ipc.Client.call(Client.java:1397)
        at org.apache.hadoop.ipc.Client.call(Client.java:1358)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
        at com.sun.proxy.$Proxy15.versionRequest(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.versionRequest(DatanodeProtocolClientSideTranslatorPB.java:272)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.retrieveNamespaceInfo(BPServiceActor.java:173)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:219)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:821)
        at java.lang.Thread.run(Thread.java:745)
Caused by: GSSException: No valid credentials provided (Mechanism level: Fail to create credential. (63) - No service crews)

Need help / pointers to fix the issue.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Services failing to restart after kerberizing cluster - HDP2.3

On the host that is showing this error, can you klist the following keytab files:

  • /etc/security/keytabs/spnego.service.keytab
  • /etc/security/keytabs/dn.service.keytab
[root@c6501 ~]# klist -kte /etc/security/keytabs/spnego.service.keytab
Keytab name: FILE:/etc/security/keytabs/spnego.service.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   1 12/11/15 15:38:21 HTTP/c6501.ambari.apache.org@EXAMPLE.COM (des-cbc-md5)
   1 12/11/15 15:38:21 HTTP/c6501.ambari.apache.org@EXAMPLE.COM (aes256-cts-hmac-sha1-96)
   1 12/11/15 15:38:21 HTTP/c6501.ambari.apache.org@EXAMPLE.COM (des3-cbc-sha1)
   1 12/11/15 15:38:21 HTTP/c6501.ambari.apache.org@EXAMPLE.COM (arcfour-hmac)
   1 12/11/15 15:38:21 HTTP/c6501.ambari.apache.org@EXAMPLE.COM (aes128-cts-hmac-sha1-96)

[root@c6501 ~]# klist -kte /etc/security/keytabs/dn.service.keytab
Keytab name: FILE:/etc/security/keytabs/dn.service.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   1 12/11/15 15:38:21 dn/c6501.ambari.apache.org@EXAMPLE.COM (des-cbc-md5)
   1 12/11/15 15:38:21 dn/c6501.ambari.apache.org@EXAMPLE.COM (aes256-cts-hmac-sha1-96)
   1 12/11/15 15:38:21 dn/c6501.ambari.apache.org@EXAMPLE.COM (des3-cbc-sha1)
   1 12/11/15 15:38:21 dn/c6501.ambari.apache.org@EXAMPLE.COM (arcfour-hmac)
   1 12/11/15 15:38:21 dn/c6501.ambari.apache.org@EXAMPLE.COM (aes128-cts-hmac-sha1-96)

Notice that my examples have entries for aes256-cts-hmac-sha1-96, do yours? If not, then you will need to make sure that you have the Unlimited Key JCE policy jars (for your JVM) installed.

If these look good, then from the host you having the issue with, can you attempt to kinit using them. If you get no messages, all is good... else there is a disconnect between your keytab file and the password stored in the KDC.

Success case:

[root@c6501 ~]# kinit -kt /etc/security/keytabs/dn.service.keytab dn/c6501.ambari.apache.org@EXAMPLE.COM
[root@c6501 ~]#

Failure case:

[root@c6501 ~]# kinit -kt /etc/security/keytabs/dn.service.keytab dn/c6501.ambari.apache.org@EXAMPLE.COM
kinit: Password incorrect while getting initial credentials
[root@c6501 ~]#

If you get this failure, then try to regenerate the keytab files from the Ambari UI to see if that helps the issue.

View solution in original post

10 REPLIES 10
Highlighted

Re: Services failing to restart after kerberizing cluster - HDP2.3

What KDC are you using? MIT? AD?

Whats your JDK version?

There could be multiple reasons for that, here are some pointers:

1)Validate the generated keytabs, this will tell you right away if there is something wrong with your keytab files or not.

kinit -kt /<path to keytabs>/<keytab file> <principal>

Check if a valid ticket was created via

klist

2) Validate JCE files: Are the JCE files available (/<jdk path>/jre/lib/security/....)? Do you need the Unlimited Strength JCEs?

3) Check permissions of the generated keytab files. For example hdfs-headless keytab should belong to hdfs:hadoop with permissions set to 0400.

4) Validate the krb5.conf file (usually under /etc/krb5.conf), make sure its available and sound.

What are the results of the above?

You might also want to read through this great guid=> https://github.com/steveloughran/kerberos_and_hado...

Highlighted

Re: Services failing to restart after kerberizing cluster - HDP2.3

Jonas

Its Cross Realm trust one way between AD and MIT kerberos in Centos. The JDK version is 1.8 and I have applied the JCE8 files . Let me validate the generated key tabs. Appreciate the suggestions.

Prijan

Highlighted

Re: Services failing to restart after kerberizing cluster - HDP2.3

@Prijan Kurup Is it MIT KDC ? or MSFT KDC? Are you able to get ticket from kdc for services like hdfs?

Highlighted

Re: Services failing to restart after kerberizing cluster - HDP2.3

Its MIT Kerberos in Centos

Highlighted

Re: Services failing to restart after kerberizing cluster - HDP2.3

On the host that is showing this error, can you klist the following keytab files:

  • /etc/security/keytabs/spnego.service.keytab
  • /etc/security/keytabs/dn.service.keytab
[root@c6501 ~]# klist -kte /etc/security/keytabs/spnego.service.keytab
Keytab name: FILE:/etc/security/keytabs/spnego.service.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   1 12/11/15 15:38:21 HTTP/c6501.ambari.apache.org@EXAMPLE.COM (des-cbc-md5)
   1 12/11/15 15:38:21 HTTP/c6501.ambari.apache.org@EXAMPLE.COM (aes256-cts-hmac-sha1-96)
   1 12/11/15 15:38:21 HTTP/c6501.ambari.apache.org@EXAMPLE.COM (des3-cbc-sha1)
   1 12/11/15 15:38:21 HTTP/c6501.ambari.apache.org@EXAMPLE.COM (arcfour-hmac)
   1 12/11/15 15:38:21 HTTP/c6501.ambari.apache.org@EXAMPLE.COM (aes128-cts-hmac-sha1-96)

[root@c6501 ~]# klist -kte /etc/security/keytabs/dn.service.keytab
Keytab name: FILE:/etc/security/keytabs/dn.service.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   1 12/11/15 15:38:21 dn/c6501.ambari.apache.org@EXAMPLE.COM (des-cbc-md5)
   1 12/11/15 15:38:21 dn/c6501.ambari.apache.org@EXAMPLE.COM (aes256-cts-hmac-sha1-96)
   1 12/11/15 15:38:21 dn/c6501.ambari.apache.org@EXAMPLE.COM (des3-cbc-sha1)
   1 12/11/15 15:38:21 dn/c6501.ambari.apache.org@EXAMPLE.COM (arcfour-hmac)
   1 12/11/15 15:38:21 dn/c6501.ambari.apache.org@EXAMPLE.COM (aes128-cts-hmac-sha1-96)

Notice that my examples have entries for aes256-cts-hmac-sha1-96, do yours? If not, then you will need to make sure that you have the Unlimited Key JCE policy jars (for your JVM) installed.

If these look good, then from the host you having the issue with, can you attempt to kinit using them. If you get no messages, all is good... else there is a disconnect between your keytab file and the password stored in the KDC.

Success case:

[root@c6501 ~]# kinit -kt /etc/security/keytabs/dn.service.keytab dn/c6501.ambari.apache.org@EXAMPLE.COM
[root@c6501 ~]#

Failure case:

[root@c6501 ~]# kinit -kt /etc/security/keytabs/dn.service.keytab dn/c6501.ambari.apache.org@EXAMPLE.COM
kinit: Password incorrect while getting initial credentials
[root@c6501 ~]#

If you get this failure, then try to regenerate the keytab files from the Ambari UI to see if that helps the issue.

View solution in original post

Highlighted

Re: Services failing to restart after kerberizing cluster - HDP2.3

[root@hashmap keytabs]# kinit -kt /etc/security/keytabs/dn.service.keytab dn/hashmap.domain.com@HADOOP.COM [root@hashmap keytabs]# klist Ticket cache: FILE:/tmp/krb5cc_0 Default principal: dn/hashmap.domain.com@HADOOP.COM Valid starting Expires Service principal 12/12/15 00:07:38 12/13/15 00:07:38 krbtgt/HADOOP.COM@HADOOP.COM renew until 12/12/15 00:07:38

Highlighted

Re: Services failing to restart after kerberizing cluster - HDP2.3

HTTP/hashmap.domain.com@HADOOP.COM (arcfour-hmac) 3 12/11/15 23:25:47 HTTP/hashmap.domain.com@HADOOP.COM (des-cbc-md5) 3 12/11/15 23:25:47 HTTP/hashmap.domain.com@HADOOP.COM (aes256-cts-hmac-sha1-96) 3 12/11/15 23:25:47 HTTP/hashmap.domain.com@HADOOP.COM (aes128-cts-hmac-sha1-96) 3 12/11/15 23:25:47 HTTP/hashmap.domain.com@HADOOP.COM (des3-cbc-sha1)

Highlighted

Re: Services failing to restart after kerberizing cluster - HDP2.3

KVNO Timestamp Principal ---- ----------------- -------------------------------------------------------- 3 12/11/15 23:25:47 dn/hashmap.domain.com@HADOOP.COM (arcfour-hmac) 3 12/11/15 23:25:47 dn/hashmap.domain.com@HADOOP.COM (des-cbc-md5) 3 12/11/15 23:25:47 dn/hashmap.domain.com@HADOOP.COM (aes256-cts-hmac-sha1-96) 3 12/11/15 23:25:47 dn/hashmap.domain.com@HADOOP.COM (aes128-cts-hmac-sha1-96) 3 12/11/15 23:25:47 dn/hashmap.domain.com@HADOOP.COM (des3-cbc-sha1)

Highlighted

Re: Services failing to restart after kerberizing cluster - HDP2.3

@Robert Levas

The following is the output

[root@hashmap keytabs]# klist -kte /etc/security/keytabs/spnego.service.keytab Keytab name: FILE:/etc/security/keytabs/spnego.service.keytab KVNO Timestamp Principal ---- ----------------- -------------------------------------------------------- 3 12/11/15 23:25:47 HTTP/hashmap.domain.com@HADOOP.COM (arcfour-hmac) 3 12/11/15 23:25:47 HTTP/hashmap.domain.com@HADOOP.COM (des-cbc-md5) 3 12/11/15 23:25:47 HTTP/hashmap.domain.com@HADOOP.COM (aes256-cts-hmac-sha1-96) 3 12/11/15 23:25:47 HTTP/hashmap.domain.com@HADOOP.COM (aes128-cts-hmac-sha1-96) 3 12/11/15 23:25:47 HTTP/hashmap.domain.com@HADOOP.COM (des3-cbc-sha1) [root@hashmap keytabs]# klist -kte /etc/security/keytabs/dn.service.keytab Keytab name: FILE:/etc/security/keytabs/dn.service.keytab KVNO Timestamp Principal ---- ----------------- -------------------------------------------------------- 3 12/11/15 23:25:47 dn/hashmap.domain.com@HADOOP.COM (arcfour-hmac) 3 12/11/15 23:25:47 dn/hashmap.domain.com@HADOOP.COM (des-cbc-md5) 3 12/11/15 23:25:47 dn/hashmap.domain.com@HADOOP.COM (aes256-cts-hmac-sha1-96) 3 12/11/15 23:25:47 dn/hashmap.domain.com@HADOOP.COM (aes128-cts-hmac-sha1-96) 3 12/11/15 23:25:47 dn/hashmap.domain.com@HADOOP.COM (des3-cbc-sha1) [root@hashmap keytabs]#

Don't have an account?
Coming from Hortonworks? Activate your account here