Kerberos: Preauthentication failed while getting initial credentials

[#1]

Dear experts,

We are enabling kerberos in our cluster with integrating it to Active Directory. The Kerberos has been enabled however during the service restarts, all the services are being failed with the below error, could you please assist on this ?

/usr/bin/kinit -kt /etc/security/keytabs/smokeuser.headless.keytab ambari-qa-hdp@HADOOP.LOCAL;' returned 1. kinit: Preauthentication failed while getting initial credentials

Zookeeper logs:

2018-06-17 14:02:52,217 - INFO [PurgeTask:DatadirCleanupManager$PurgeTask@138] - Purge task started. 2018-06-17 14:02:52,218 - INFO [main:QuorumPeerMain@127] - Starting quorum peer 2018-06-17 14:02:52,229 - INFO [PurgeTask:DatadirCleanupManager$PurgeTask@144] - Purge task completed. 2018-06-17 14:02:52,814 - ERROR [main:QuorumPeerMain@89] - Unexpected exception, exiting abnormally java.io.IOException: Could not configure server because SASL configuration did not allow the ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: Pre-authentication information was invalid (24) at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:207) at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:87) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:130) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)

Because of this issues, it is not allowing us to disable the kerberos now. Kindly help on this.

Thanks,

Chiranjeevi

[#2]

This preauthentication failure can happen for several reasons. Mostly we see when either the password for the relevant account in the Active Directory has changed since the keytab file was created; or the system clock is off by about 5 minutes from that of the Active Directory.

Is it possible one of these 2 scenarios are in play?

For that Zookeeper issue, I think you may be able to manually start Zookeeper in a permissive mode so that you can manually update the ACLs on the znodes. However I am not too familiar with this part of the equation.

[#3]

@rlevas what if the password has changed? what steps do we need to take as a fix?