Failing to connect to KDC during enable kerberos, CA certificate has been imported into Ambari & Java trust stores

[#1]

Hi there good folks

We are trying to enable HDP kerberos integration, but we are getting stuck in the Wizard during "test kerberos client".

Failed to connect to KDC - Failed to communicate with the Active Directory at ldaps://ad-serverxxxx:636: simple bind failed: ad-serverxxxx:636
Make sure the server's SSL certificate or CA certificates have been imported into Ambari's truststore.

Verified both JAVA and AMBARI CA certs in Trust Stores.

$JAVA_HOME/bin/keytool -list -v -keystore $JAVA_HOME/lib/security/cacerts > /tmp/05122018_java_truststore_cert

--Did the same writing out the Ambari trust store cert.

The certs are there and confirmed not yet expired.

Next try to test the service account used and ensure the accounts works fine:

ldapsearch -x -LLL -h ad-serverxxxxx -D 'CN=S_LDAP_HortonWrks_DEV,OU=Admin,OU=xxx,DC=xxxxxxx,DC=xxx,DC=xxx' -b "OU=HDP,DC=xxx,DC=xxx,DC=xxx" -W

Queries for password , authenticates and returned successfully so the account seems fine.

The irony is that we did this just a few weeks before and didn't have issues but had to tear down and rebuild due to another un-related issue.

Last time we got stuck at the same place but then import the DC's cert into the JAVA cacerts trust store resolved the issue.

Now for some reason it's not. The master is a clean new server, the slaves are the old machines that have been cleared up using this blog.

https://community.hortonworks.com/articles/97489/completely-uninstall-hdp-and-ambari.html

Any help would be highly appreciated. Drawing a bit of a blank after all the troubleshooting done so far.

Kind Regards

[#2]

@Nico Jordaan Take a look at the Ambari server log (/var/log/ambari-server/ambari-server.log) to see what the complete issue is. It could be related to the SSL cert or maybe the truststore you think it being used by Ambari is not really the truststore that Ambari is using.

One thing you can try is to disable certificate validation when connecting to the Active Directory while enabling Kerberos. This can be done by setting the following property in the ambari.properties file:

kerberos.operation.verify.kdc.trust = true

Then restart Ambari and try to re-enable Kerberos.

[#3]

I have check the Ambari Server log but it's not really very helpful.

sudo $JAVA_HOME/bin/keytool -list -v -keystore /var/lib/ambari-server/keys/ambari-server-truststore > /tmp/05122018_Ambari_truststore_cert

Confirmed the truststore location matches the ambari.properties location under /etc/ambari-server/conf/ambari.properties.

@Robert Levas , Your suggestion might help but do you not recon it might cause issues later down the line? Feels like it would be a bit "hacky" 🙂 .. Kind Regards

05 Dec 2018 14:56:00,217 ERROR [ambari-client-thread-303] KerberosHelperImpl:2232 - Cannot validate credentials: org.apache.ambari.server.serveraction.kerberos.KerberosInvalidConfigurationException: Failed to connect to KDC - Failed to communicate with the Active Directory at ldaps://ad-serverxxxx:636: simple bind failed ad-serverxxxx:636 Make sure the server's SSL certificate or CA certificates have been imported into Ambari's truststore. 05 Dec 2018 14:56:00,217 ERROR [ambari-client-thread-303] BaseManagementHandler:67 - Bad request received: Failed to connect to KDC - Failed to communicate with the Active Directory at ldaps://ad-serverxxxx:636 simple bind failed: ad-serverxxx:636 Make sure the server's SSL certificate or CA certificates have been imported into Ambari's truststore. 05 Dec 2018 15:02:51,205 INFO [ambari-client-thread-554] AmbariManagementControllerImpl:4173 - Received action execution request, clusterName=caphdpoc, request=isCommand :true, action :null, command :KERBEROS_SERVICE_CHECK, inputs :{HAS_RESOURCE_FILTERS=true}, resourceFilters: [RequestResourceFilter{serviceName='KERBEROS', componentName='null', hostNames=[]}], exclusive: false, clusterName :caphdppoc 05 Dec 2018 15:02:51,364 WARN [ambari-client-thread-554] ADKerberosOperationHandler:470 - Failed to communicate with the Active Directory at ldaps://ad-serverxxxx:636:: simple bind failed: ad-serverxxxx:636 javax.naming.CommunicationException: simple bind failed: ad-serverxxxx:636 [Root exception is javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target] at com.sun.jndi.ldap.LdapClient.authenticate(LdapClient.java:219) at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2791) at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:319) at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:192) at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:210) at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:153) at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:83) at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684) at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:313) at javax.naming.InitialContext.init(InitialContext.java:244) at javax.naming.ldap.InitialLdapContext.<init>(InitialLdapContext.java:154) at org.apache.ambari.server.serveraction.kerberos.ADKerberosOperationHandler.createInitialLdapContext(ADKerberosOperationHandler.java:514) at org.apache.ambari.server.serveraction.kerberos.ADKerberosOperationHandler.createLdapContext(ADKerberosOperationHandler.java:465) at org.apache.ambari.server.serveraction.kerberos.ADKerberosOperationHandler.open(ADKerberosOperationHandler.java:182) at ......

com.sun.jndi.ldap.Connection.writeRequest(Connection.java:416) at com.sun.jndi.ldap.LdapClient.ldapBind(LdapClient.java:359) at com.sun.jndi.ldap.LdapClient.authenticate(LdapClient.java:214) ... 114 more Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) at sun.security.validator.Validator.validate(Validator.java:262) at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621) ... 127 more Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) ... 133 more 05 Dec 2018 15:02:51,367 ERROR [ambari-client-thread-554] KerberosHelperImpl:2232 - Cannot validate credentials: org.apache.ambari.server.serveraction.kerberos.KerberosInvalidConfigurationException: Failed to connect to KDC - Failed to communicate with the Active Directory at ldaps://ad-serverxxxx:636 simple bind failed: ad-serverxxxx:636 Make sure the server's SSL certificate or CA certificates have been imported into Ambari's truststore. 05 Dec 2018 15:02:51,367 ERROR [ambari-client-thread-554] BaseManagementHandler:67 - Bad request received: Failed to connect to KDC - Failed to communicate with the Active Directory at ldap://ad-serverxxxx:636: simple bind failed: ad-serverxxxx:636 Make sure the server's SSL certificate or CA certificates have been imported into Ambari's truststore.

[#4]

My suggestion to set kerberos.operation.verify.kdc.trust to true is a bit of a hack, but it may give you an idea of what the cause is. If this works, then there is something up with the Ambari trust store... like the needed CA certs have not been imported, or maybe Ambari is not really using the one you think it is. Once we figure out a solution to the issue, we can flip the flag back to true (or remove that property) and you will have SSL certificate trust validation turned on again.

Looking at the log entries, the issue points to a lack of information in the trust store :

sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Did you try adding all CA and intermediary CA certs into the trust store?