Some services failing to come up after enabling kerberos

[#1]

Error:

at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:302)
at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:120)
at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458)
at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693)
... 43 more
Caused by: KrbException: Identifier doesn't match expected value (906)
at sun.security.krb5.internal.KDCRep.init(KDCRep.java:140)
at sun.security.krb5.internal.TGSRep.init(TGSRep.java:65)
at sun.security.krb5.internal.TGSRep.<init>(TGSRep.java:60)
at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:55)
... 49 more
16/11/29 13:13:12 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.

Cause:

The cause for this issue was that there were multiple accounts in the Active Directory that had a servicePrincipalName value containing the Zookeeper principal names - "zookeeper/<hostname>".

This was found by issuing an ldapsearch like:

ldapsearch -h <host> -D <user principal> -W -b "<bind dn - something high in the tree>" '(servicePrincipalName=zookeeper/<zk server hostname>)' dn 

This request found 2 accounts that contained the requested SPN. One way to know this may be an issue is after authenticating (kinit-ing) as any valid user, issue a kvno command like kvno zookeeper/abc.ambari.apache.org If this fails but a different service principal (like nn/abc.ambari.apache.org) succeeds, then the above cause may be the problem.

Solution:

Find all duplicated SPN values and remove the non-Ambari-managed ones from the Active Directory. Then restart all of the services. Optionally all of the Keytab files can be regenerated to make sure all is in a good state.