Zookeeper service fails to start with checksum error after Kerberos is enabled

[#1]

Zookeeper service fails to start after kerberos has been enabled with the following error -

ERROR org.apache.zookeeper.server.quorum.QuorumPeerMain: Unexpected exception, exiting abnormally
java.io.IOException: Could not configure server because SASL configuration did not allow the ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: Checksum failed
at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:207)
at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:87)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:135)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)

please advise. I have been through multiple posts but do not see any solution regarding this.

[#2 · Accepted solution]

@Krish216,

Do you have any other stack information or is that all that appears?

The failure there is occurring when attempting to get a TGT from your KDC.

The "checksum" error likely occurs after the KDC has replied to the AS_REQ (TGT request) because the reply cannot be decrypted.

It could be that your krb5.conf file has encryption types listed that are not in the zookeeper keytab.

Recommendations:

1. make certain that your /etc/krb5.conf on the zookeeper host contains only the encryption types that are in the zookeeper keytab

2.  If (1) does not help or is not the issue, try regenerating the zookeeper credentials in Cloudera Manager to ensure that your keytab contains the same keys as the KDC for that principal.

[#3]

This is obviously an issue with the Kerberos. Checksum error isually when keytab is not created correctly. Have you manually created and distributed keytabs? If yes, then check the integrity of keytab by trying to authenticate using it (kinit).

If you rely on Cloudera Manager (which is preferable) to do this job, then you should check the zookeeper node. E.g.:

* /etc/krb5.conf is correct

* you can authenticate to Kerberos from this node

* Java Cryptography Extension (JCE) is installed and zookeeper is launched with the correct java.

[#4]

Thanks Gekas for the suggestions. I am able to do kinit with the keytab that is generated without any issue.

While starting zookeeper is taking JAVA_HOME as /usr/java/jdk1.8.0 and I have installed the JCE files at the /usr/java/jdk1.8.0/jre/lib/security. There is only one version of java that is installed. I did a netcat from source to KDC hosts and those are working fine as well.

Not sure If I'm missing anything. 

[#5 · Accepted solution]

@Krish216,

Do you have any other stack information or is that all that appears?

The failure there is occurring when attempting to get a TGT from your KDC.

The "checksum" error likely occurs after the KDC has replied to the AS_REQ (TGT request) because the reply cannot be decrypted.

It could be that your krb5.conf file has encryption types listed that are not in the zookeeper keytab.

Recommendations:

1. make certain that your /etc/krb5.conf on the zookeeper host contains only the encryption types that are in the zookeeper keytab

2.  If (1) does not help or is not the issue, try regenerating the zookeeper credentials in Cloudera Manager to ensure that your keytab contains the same keys as the KDC for that principal.

[#6 · Accepted solution]

@bgooley

Here is what happened -  KDC slave that we're connecting was not running the kpropd demon. Once started we're able to pass the issue.

Also, checking on encryption types in kdc.conf made the issue completely resolved. Thank you all the help & suggestions.

[#7]

Could you please mark it as answered so the community will be benifited ?