SSL handshake error when configuring SSL for Cloudera Navigator (cloudera-navigator.properties)

[#1]

After having succesfully enabled TLS encryption between Server and Agents, I am unable to load Cloudera Navigator UI. The log is pointing at issues with SSL handshake.

I understand I need to configure SSL for Cloudera Navigator in addition to this, so I followed guidelines from Cloudera documentation:

1. Open the Cloudera Manager Admin Console and navigate to the Cloudera Management Service.
2. Click Configuration.
3. Go to the Navigator Metadata Server Default Group > Advanced category, and add the following strings to the Navigator Metadata Server Advanced Configuration Snippet (Safety Valve) for cloudera-navigator.properties property.
   nav.http.enable_ssl=true
   nav.ssl.keyStore=<path to jks keystore with signed server certificate installed>
   nav.ssl.keyStorePassword=<password>
4. Click Save Changes.
5. Restart the Navigator Metadata server.

After I added cloudera-navigator.properties to Safety Valve and restarted, Cloudera Management Services became unhealthy and I had to revert my change. I would like to clarify what values exactly go into nav.ssl.keyStore and nav.ssl.keyStorePassword. I have set nav.ssl.keyStore to same value as ssl.client.truststore.location, since this is where my keystore file lives. 

2015-04-15 17:54:02,572 WARN com.cloudera.enterprise.EnterpriseService: Exception in scheduled runnable.
javax.ws.rs.client.ClientException: org.apache.cxf.interceptor.Fault: Could not send Message.
    at org.apache.cxf.jaxrs.client.AbstractClient.checkClientException(AbstractClient.java:548)
    at org.apache.cxf.jaxrs.client.AbstractClient.preProcessResult(AbstractClient.java:534)
    at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:545)
    at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:206)
    at com.sun.proxy.$Proxy35.readRoles(Unknown Source)
    at com.cloudera.nav.cm.CmApiClient.getMgmtRoleByType(CmApiClient.java:224)
    at com.cloudera.navigator.ActivityPollingService.getAmonNozzle(ActivityPollingService.java:189)
    at com.cloudera.navigator.ActivityPollingService.run(ActivityPollingService.java:108)
    at com.cloudera.enterprise.PeriodicEnterpriseService$UnexceptionablePeriodicRunnable.run(PeriodicEnterpriseService.java:67)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.cxf.interceptor.Fault: Could not send Message.
    at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:64)
    at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271)
    at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:607)
    at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:543)
    ... 7 more
Caused by: javax.net.ssl.SSLHandshakeException: SSLHandshakeException invoking https://{HOSTNAME}:7183/api/v4/cm/service/roles: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.reflect.GeneratedConstructorAccessor51.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1338)
    at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1322)
    at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
    at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:622)
    at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
    ... 10 more
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1884)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:276)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:270)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1341)
    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:153)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
    at sun.security.ssl.Handshaker.process_record(Handshaker.java:804)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1016)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323)
    at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:563)
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.jav...
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1300)
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
    at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.getResponseCode(URLConnectionHTTPConduit.java:260)
    at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1517)
    at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1490)
    at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1309)
    ... 13 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
    at sun.security.validator.Validator.validate(Validator.java:260)
    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:326)
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231)
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:126)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1323)
    ... 29 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:196)
    at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:268)
    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:380)
    ... 35 more

[#2]

Troubleshooting SSL/TLS Connectivity. Verified connectivity.

**{HOSTNAME} refers to the hostname listed in the logs, where Cloudera Manager Server lives

# openssl s_client -connect {HOSTNAME}:7183
CONNECTED(00000003)
depth=1 O = PLATFORM.{OUR_DOMAIN], CN = Certificate Authority
verify error:num=19:self signed certificate in certificate chain
verify return:0
---
Certificate chain
 0 s:/O=PLATFORM.{OUR_DOMAIN]/CN={HOSTNAME}
   i:/O=PLATFORM.{OUR_DOMAIN]/CN=Certificate Authority
 1 s:/O=PLATFORM.{OUR_DOMAIN]/CN=Certificate Authority
   i:/O=PLATFORM.{OUR_DOMAIN]/CN=Certificate Authority
---
Server certificate
-----BEGIN CERTIFICATE-----

...here goes our certificate

-----END CERTIFICATE-----
subject=/O=PLATFORM.{OUR_DOMAIN]/CN={HOSTNAME}
issuer=/O=PLATFORM.{OUR_DOMAIN]/CN=Certificate Authority
---
No client certificate CA names sent
Server Temp Key: ECDH, ___, 521 bits
---
SSL handshake has read 2508 bytes and written 511 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ___
    Session-ID: ___
    Session-ID-ctx:
    Master-Key: ___
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1429155044
    Timeout   : 300 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
---

[#3]

Have you established implicit trust for your private CA in the JDK layer as discussed in our documentation on encryption for the platform, as laid out here:

http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_sg_create_key_trust....

You can import the CA's certificate into the [JAVA_HOME]/jre/lib/security path, using cacerts or jssecacerts as you see fit.

Trust in the JDK is documented in detail here as well

http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html#X509TrustManag...

[#4]

So far it appears that only Navigator is unhappy with the keystore. So I believe TLS/SSL was set up correctly, otherwise. We are using FreeIPA as certificate authority and below is a quick overview of steps taken to set it up, since there is a somewhat deviation from standard protocol.

On the Namenode (same host the Cloudera Manager lives on), I generated a certificate and key to be used by Cloudera Manager

# kinit -kt /etc/krb5.keytab

# ipa-getcert request -f cmhost.pem -k cmhost.key -r

# chmod 600 cmhost*

Then I copied the newly created cm-keys directory to each host.

$ for x in {LIST_OF_CDH_HOSTS}; do scp -r cm-keys $x:; done

$ for x in {LIST_OF_CDH_HOSTS}; do ssh -tty $x sudo bash -c "'mkdir -p /opt/cloudera/security/x509; mv cm-keys/* /opt/cloudera/security/x509; chown cloudera /opt/cloudera/security/x509/*'"; done

Next, I set up Puppet to configure Cloudera to use TLS.

[#5]

Ok so you have 2 paradigims that you are configuring for, the JDK, and OpenSSL

"Trust" is established differently between the two implementations, Navigator, being Java based, will derive trust through the default JDK mechanisims I pointed out

For the Agents, Hue, and Implala the implementation is based on OpenSSL. Trust can be established for a root CA by directly configuring path to a CA pem file, or if there is a complex intermediary chain through providing the path to where the files are and running the c_rehash tool to generate the necessary base64 encoded symlinks.

Navigator is expecting you to configure a jks with private key and installed cert, but then as navigator attempts to connect to other cluster services that are using SSL/TLS, it needs to trust the CA that issued the server cert...

[#6]

"Trust" is established differently between the two implementations, Navigator, being Java based, will derive trust through the default JDK mechanisms I pointed out

We are looking into whether we need to use keytool utility to generate those or use our FreeIPA server to generate certs for Navigator...

[#7]

And we cover import/export scenarios between the two implementations here:

http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_sg_openssl_jks.html

So you need to consider that you have a private key component, and a issued certificate component... the combination of the two in a JKS (private key and certificate) allow you to start up a java based service that will support SSL/TLS connections.

You can use your IPA based appoach as long as you have access to the private key and certificate it issues, to combine them and then import into a JKS.

The the CA certificate that issued the certificate, imported into the truststores I've discussed already, establishes inherent trust within java services, for SSL certificates created by that CA.

[#8]

The the CA certificate that issued the certificate, imported into the truststores I've discussed already, establishes inherent trust within java services, for SSL certificates created by that CA.

We have already created a truststore on Namenode (node where Cloudera Manager is installed), when TLS was set up for all agents, that's what was used - /etc/cloudera-scm-server/keystore - and this file was copied to all nodes in Cloudera Hadoop cluster, including Navigator.

[#9 · Accepted solution]

Sorry,

So the term "truststore" is over-loaded.  Are you saying a JKS file that you configure from the CM UI as a truststore for each service in the cluster, including management services?  Or a "default" truststore like [JAVA_HOME]/jre/lib/security/cacerts (or jssecacerts) that establishes inherent trust as we have been discussing...

You have a number of configuration changes to make, one for each service, to recognize that trust store file (as opposed to instrumenting the JDK for trust).

[#10]

Thank you! We got this to work finally. Now just need to wire to LDAP.

It looks like the keystore passed by following the instructions on Cloudera's site wasn't used for some reason. Following instructions here: http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_sg_create_key_trust....

The upshot is that the keystore at /usr/java/jdk1.7.0_67-cloudera/jre/lib/security/jssecacerts (or ${JAVA_HOME}/lib/security/jssecacerts, where $JAVA_HOME is the home of the version of Java used by Cloudera Navigator, we used "ps" to find out where) should contain the root certificate. We then restarted both Cloudera Navigator services, and able to navigate to https://

Integration with FreeIPA was sort of confusing initially. Thanks for your help in understanding the mechnisms of this functionality.