How to solve the `Message stream modified (41)` error on starting Kerberized components.
What happened?
Starting up a ZooKeeper server in a Kerberized CDP-DC 7.0.3 cluster failed with the logs below.
2020-03-30 12:23:10,251 ERROR org.apache.zookeeper.server.quorum.QuorumPeerMain: Unexpected exception, exiting abnormally
java.io.IOException: Could not configure server because SASL configuration did not allow the ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: Message stream modified (41)
at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:243)
at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:148)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
The JDK for this environment is OpenJDK 1.8.0_242.
# java -version
openjdk version "1.8.0_242"
OpenJDK Runtime Environment (build 1.8.0_242-b08)
OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)
Solution
Removing the line of renew_lifetime in /etc/krb5.conf.
Removing this line means to use the default value, 0, for renew_lifetime.
Thus, it may also need to specify renew_lifetime when running kinit command.
See also
On this page, Akira Ajisaka, one of Hadoop PMCs, described the solution.
He also mentioned a related OpenJDK's JIRA ticket.
https://bugs.centos.org/view.php?id=17000
This page also introduced the same solution.
Additionally, this page showed another solution, setting sun.security.krb5.disableReferrals=true in java.security file. But in my case, this solution didn't work.
This is a related article from the Cloudera Knowledge Base.
It also describes sun.security.krb5.disableReferrals=true as its workaround.
