... 43 more
Caused by: KrbException: Identifier doesn't match expected value (906)
... 49 more
16/11/29 13:13:12 WARN security.UserGroupInformation: Not attempting to re-login since the last re-login was attempted less than 600 seconds before.
The cause for this issue was that there were multiple accounts in the Active Directory that had a servicePrincipalName value containing the Zookeeper principal names - "zookeeper/<hostname>".
This was found by issuing an ldapsearch like:
ldapsearch -h <host> -D <user principal> -W -b "<bind dn - something high in the tree>" '(servicePrincipalName=zookeeper/<zk server hostname>)' dn
This request found 2 accounts that contained the requested SPN. One way to know this may be an issue is after authenticating (kinit-ing) as any valid user, issue a kvno command like kvno zookeeper/abc.ambari.apache.org If this fails but a different service principal (like nn/abc.ambari.apache.org) succeeds, then the above cause may be the problem.
Find all duplicated SPN values and remove the non-Ambari-managed ones from the Active Directory. Then restart all of the services. Optionally all of the Keytab files can be regenerated to make sure all is in a good state.