Created on
03-31-2020
05:39 AM
- edited on
11-19-2020
04:38 AM
by
VidyaSargur
Starting up a ZooKeeper server in a Kerberized CDP-DC 7.0.3 cluster failed with the logs below.
2020-03-30 12:23:10,251 ERROR org.apache.zookeeper.server.quorum.QuorumPeerMain: Unexpected exception, exiting abnormally
java.io.IOException: Could not configure server because SASL configuration did not allow the ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: Message stream modified (41)
at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:243)
at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:148)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
The JDK for this environment is OpenJDK 1.8.0_242.
# java -version
openjdk version "1.8.0_242"
OpenJDK Runtime Environment (build 1.8.0_242-b08)
OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)
Removing the line of renew_lifetime in /etc/krb5.conf.
Removing this line means to use the default value, 0, for renew_lifetime.
Thus, it may also need to specify renew_lifetime when running kinit command.
On this page, Akira Ajisaka, one of Hadoop PMCs, described the solution.
He also mentioned a related OpenJDK's JIRA ticket.
https://bugs.centos.org/view.php?id=17000
This page also introduced the same solution.
Additionally, this page showed another solution, setting sun.security.krb5.disableReferrals=true in java.security file. But in my case, this solution didn't work.
This is a related article from the Cloudera Knowledge Base.
It also describes sun.security.krb5.disableReferrals=true as its workaround.
Created on 11-18-2020 04:06 PM - edited 11-18-2020 04:41 PM
Thanks for the solution!! Same issue for me after enabling MIT Kerberos in the CDH 5.16.2 cluster zookeeper wouldn't start with the above message
javax.security.auth.login.LoginException: Message stream modified (41)
I was using openjdk version "1.8.0_272". As per your solution commented the line in /etc/krb5.conf on all servers:
#renew_lifetime = 604800
After that restart of cluster all services worked except Hue Kerberos Ticket Renewer which gives error
Couldn't renew kerberos ticket in order to work around Kerberos 1.8.1 issue. Please check that the ticket for 'hue/fqdn@KRBREALM' is still renewable:
The Kerberos Ticket Renewer is a separate issue and we need to run on the MIT KDC server:
kadmin.local: modprinc -maxrenewlife 90day krbtgt/KRBREALM
kadmin.local: modprinc -maxrenewlife 90day +allow_renewable hue/fqdn@KRBREALM for all hue servers fqdn
After that Hue Kerberos Ticket Renewer restarted successfully.