How to solve the `Message stream modified (41)` error on starting Kerberized components.

[#1]

What happened?

Starting up a ZooKeeper server in a Kerberized CDP-DC 7.0.3 cluster failed with the logs below. 

2020-03-30 12:23:10,251 ERROR org.apache.zookeeper.server.quorum.QuorumPeerMain: Unexpected exception, exiting abnormally
java.io.IOException: Could not configure server because SASL configuration did not allow the  ZooKeeper server to authenticate itself properly: javax.security.auth.login.LoginException: Message stream modified (41)
        at org.apache.zookeeper.server.ServerCnxnFactory.configureSaslLogin(ServerCnxnFactory.java:243)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:646)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:148)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:123)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)

The JDK for this environment is OpenJDK 1.8.0_242.

# java -version
openjdk version "1.8.0_242"
OpenJDK Runtime Environment (build 1.8.0_242-b08)
OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)

Solution

Removing the line of renew_lifetime in /etc/krb5.conf.

Removing this line means to use the default value, 0, for renew_lifetime.

Thus, it may also need to specify renew_lifetime when running kinit command.

See also

http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201909.mbox/%3CCAKRKJ1O3yrYKDZ=WhU=i6A+zqxF...

On this page, Akira Ajisaka, one of Hadoop PMCs, described the solution.

He also mentioned a related OpenJDK's JIRA ticket.

https://bugs.centos.org/view.php?id=17000

This page also introduced the same solution.

Additionally, this page showed another solution, setting sun.security.krb5.disableReferrals=true in java.security file. But in my case, this solution didn't work.

https://my.cloudera.com/knowledge/Cloudera-Customer-Advisory-Servers-with-Kerberos-enabled-stop?id=2...

This is a related article from the Cloudera Knowledge Base.

It also describes sun.security.krb5.disableReferrals=true as its workaround.

[#2]

Thanks for the solution!! Same issue for me after enabling MIT Kerberos in the CDH 5.16.2 cluster zookeeper wouldn't start with the above message

javax.security.auth.login.LoginException: Message stream modified (41)

I was using openjdk version "1.8.0_272". As per your solution commented the line in /etc/krb5.conf on all servers:

#renew_lifetime = 604800    

After that restart of cluster all services worked except Hue Kerberos Ticket Renewer which gives error

Couldn't renew kerberos ticket in order to work around Kerberos 1.8.1 issue. Please check that the ticket for 'hue/fqdn@KRBREALM' is still renewable:

The Kerberos Ticket Renewer is a separate issue and we need to run on the MIT KDC server:

kadmin.local:  modprinc -maxrenewlife 90day krbtgt/KRBREALM

kadmin.local:  modprinc -maxrenewlife 90day +allow_renewable hue/fqdn@KRBREALM   for all hue servers fqdn

After that  Hue Kerberos Ticket Renewer restarted successfully.