MIT Kerberos to Active Directory migration issue

[#1]

Hello Experts,

After changing MIT Kerberos to AD Kerberos and Regenerating all the Kerberos credentials in CM the zookeeper, YARN etc. is not starting. There is an error about the Active Directory samaccount not able to login as the zookeeper principal. I checked that the principals are created in the AD OrgUnit for Cloudera. And the $ kinit -kt zookeeper.keytab zookeeper/redacted@ADREALM on the linux servers works fine.  

Any thoughts how to fix?

SERVICE_TYPEZOOKEEPER SEVERITYCRITICAL 

STACKTRACE

javax.security.sasl.SaslException: Problem with callback handler [Caused by javax.security.sasl.SaslException: redacted@ADREALM.COM is not authorized to connect as zookeeper/redacted@ADREALM.COM] at com.sun.security.sasl.gsskerb.GssKrb5Server.doHandshake2(GssKrb5Server.java:333) at com.sun.security.sasl.gsskerb.GssKrb5Server.evaluateResponse(GssKrb5Server.java:161) at org.apache.zookeeper.server.quorum.auth.SaslQuorumAuthServer.authenticate(SaslQuorumAuthServer.java:98) at org.apache.zookeeper.server.quorum.QuorumCnxManager.handleConnection(QuorumCnxManager.java:449) at org.apache.zookeeper.server.quorum.QuorumCnxManager.receiveConnection(QuorumCnxManager.java:387) at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReceiverThread.run(QuorumCnxManager.java:423) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: javax.security.sasl.SaslException: 

Thanks!

[#2 · Accepted solution]

@ebeb From the logs seems like you are hitting a Java bug. This happens after upgrading the JDK version, on Open JDK versions after JDK8u262 this is due to: https://bugs.openjdk.java.net/browse/JDK-8230496

Resolution: 

Edit java.security file located in the active JDK on the clusters.

Add or alter sun.security.krb5.disableReferrals parameter, to ensure that the following is set to true:
sun.security.krb5.disableReferrals=true

You may find the Active JDK version being used by CDH by using the bigtop script to load the JAVA_HOME
source /opt/cloudera/parcels/CDH/bin/bigtop-detect-javahome
echo $JAVA_HOME
echo "sun.security.krb5.disableReferrals=true" >> $JAVA_HOME/jre/lib/security/java.security

Restart the service. 

[#3]

@GangWar you are a genius! After this java parameter change all CDH services started smoothly and everything running fine with Active Directory kerberos. Thanks so much!

[#4]

Can you please help me on how can I migrate from MIT kerberos to AD kerberos if currently MIT kerberos is being used by 6000+ applications, or can you share some documentation on how to do it?