Zookeeper unable to start after enabling Kerberos , SASL error

[#1]

Hi to all,

My cluster is the latest CM 7.4.4, and CR 7.1.7.

Cluster working fine until I enable Kerberos. Zookeeper wont start with error 

Could not configure server because SASL configuration did not allow the Zookeeper server to authenticat itself properly:javax.security.auth.login.LoginException: Message stream modified (41)

I'm able to get zookeeper and other services up if I commented

# renew_lifetime = 7d  on all the nodes and kerberos server.

But only Hue Kerberos Ticket Renewer will have a problem. So what I did was I commented out 

renew_lifetime = 7d on server that hosted Kerberos Ticket REnewar roles.

So now my cluster will be up. But this does not like a good workarund as some of the UI are having a problem like Atlas and Solr with error (tgt renewal).

Anyone encounter this?

P/S: I have a working Kerberized cluster with same version of CDP. It is working fine. Same exact version, os version, java version, and kerberos version. Only not all components is available in this cluster. So weird. 

[#2]

@muslihuddin You are running into a known Java Bug which has been found Earlier and documented. You have to follow below steps to overcome this issue..

For JDK 8u241 and higher versions running on Kerberized clusters, you must disable referrals by settingsun.security.krb5.disableReferrals=true.

For example, with OpenJDK 1.8.0u242:

1. Open /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el7_7.x86_64/jre/lib/security/java.security with a text editor.
2. Add sun.security.krb5.disableReferrals=true (it can be at the bottom of the file).
3. Add this property on each node that has the impacted JDK version.
4. Restart the applications using the JDK so the change takes effect.

For more information, see the KB article. You can so many similar discussion on Cloudera Community which has been resolved Earlier. 
[1] https://docs.cloudera.com/cdp-private-cloud-upgrade/latest/release-guide/topics/cdpdc-java-requireme...

[#3]

Hi @GangWar 

Thank you very much for your response. 

I did suspect something got to do with my java version and I've already did what you mention to disable the referrals setting sun.security.krb5.disableReferrals=true but the zookeeper still unable to start.

On the problematic cluster, I'm using OpenJDK 1.8.0u262. I have one more kerberized cluster that is running fine using OpenJDK 1.8.0u312.So what other things I tried previously. 

  1. Downgraded my OpenJDK to match the problematic version u262.

  2. Restarted cluster few times.

  3. Cluster still working fine with Kerberos, no need to comment renew_lifetime

That is why I ignored the Java version suspicion.

So the only thing for now that can make my zookeeper start was by commenting on the renew_lifetime. This guy have the same exact thing with my problem and solution. He did try the referrals as well. Do you think there are any other bugs related to this problem?

https://community.cloudera.com/t5/Community-Articles/How-to-solve-the-Message-stream-modified-41-err...

Thank you and regards.

Mus

[#4]

@muslihuddin No I didn’t find any other bug. Not sure in your case by the modification of java.security file didn’t worked alone. The solution you are having atm is also fine in my opinion no harm in that.