question Re: SSL handshake error when configuring SSL for Cloudera Navigator (cloudera-navigator.properties) in Support Questions

SSL handshake error when configuring SSL for Cloudera Navigator (cloudera-navigator.properties)

nataliaking_csc — Fri, 16 Sep 2022 09:26:34 GMT

After having succesfully enabled TLS encryption between Server and Agents, I am unable to load Cloudera Navigator UI. The log is pointing at issues with SSL handshake.

I understand I need to configure SSL for Cloudera Navigator in addition to this, so I followed guidelines from Cloudera documentation:

Open the Cloudera Manager Admin Console and navigate to the Cloudera Management Service.
Click Configuration.
Go to the Navigator Metadata Server Default Group > Advanced category, and add the following strings to the Navigator Metadata Server Advanced Configuration Snippet (Safety Valve) for cloudera-navigator.properties property.
nav.http.enable_ssl=true
nav.ssl.keyStore=<path to jks keystore with signed server certificate installed>
nav.ssl.keyStorePassword=<password>
Click Save Changes.
Restart the Navigator Metadata server.

After I added cloudera-navigator.properties to Safety Valve and restarted, Cloudera Management Services became unhealthy and I had to revert my change. I would like to clarify what values exactly go into nav.ssl.keyStore and nav.ssl.keyStorePassword. I have set nav.ssl.keyStore to same value as ssl.client.truststore.location, since this is where my keystore file lives.

2015-04-15 17:54:02,572 WARN com.cloudera.enterprise.EnterpriseService: Exception in scheduled runnable.
javax.ws.rs.client.ClientException: org.apache.cxf.interceptor.Fault: Could not send Message.
   at org.apache.cxf.jaxrs.client.AbstractClient.checkClientException(AbstractClient.java:548)
   at org.apache.cxf.jaxrs.client.AbstractClient.preProcessResult(AbstractClient.java:534)
   at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:545)
   at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:206)
   at com.sun.proxy.$Proxy35.readRoles(Unknown Source)
   at com.cloudera.nav.cm.CmApiClient.getMgmtRoleByType(CmApiClient.java:224)
   at com.cloudera.navigator.ActivityPollingService.getAmonNozzle(ActivityPollingService.java:189)
   at com.cloudera.navigator.ActivityPollingService.run(ActivityPollingService.java:108)
   at com.cloudera.enterprise.PeriodicEnterpriseService$UnexceptionablePeriodicRunnable.run(PeriodicEnterpriseService.java:67)
   at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.cxf.interceptor.Fault: Could not send Message.
   at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:64)
   at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271)
   at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:607)
   at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:543)
   ... 7 more
Caused by: javax.net.ssl.SSLHandshakeException: SSLHandshakeException invoking https://{HOSTNAME}:7183/api/v4/cm/service/roles: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
   at sun.reflect.GeneratedConstructorAccessor51.newInstance(Unknown Source)
   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
   at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
   at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1338)
   at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1322)
   at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
   at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:622)
   at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
   ... 10 more
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
   at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
   at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1884)
   at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:276)
   at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:270)
   at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1341)
   at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:153)
   at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
   at sun.security.ssl.Handshaker.process_record(Handshaker.java:804)
   at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1016)
   at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
   at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
   at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323)
   at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:563)
   at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
   at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1300)
   at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
   at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:338)
   at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.getResponseCode(URLConnectionHTTPConduit.java:260)
   at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1517)
   at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1490)
   at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1309)
   ... 13 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
   at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385)
   at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
   at sun.security.validator.Validator.validate(Validator.java:260)
   at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:326)
   at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231)
   at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:126)
   at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1323)
   ... 29 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
   at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:196)
   at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:268)
   at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:380)
   ... 35 more

Re: SSL handshake error when configuring SSL for Cloudera Navigator (cloudera-navigator.properties)

nataliaking_csc — Thu, 18 Jun 2015 15:32:43 GMT

Troubleshooting SSL/TLS Connectivity. Verified connectivity.

**{HOSTNAME} refers to the hostname listed in the logs, where Cloudera Manager Server lives

# openssl s_client -connect {HOSTNAME}:7183
CONNECTED(00000003)
depth=1 O = PLATFORM.{OUR_DOMAIN], CN = Certificate Authority
verify error:num=19:self signed certificate in certificate chain
verify return:0
---
Certificate chain
0 s:/O=PLATFORM.{OUR_DOMAIN]/CN={HOSTNAME}
i:/O=PLATFORM.{OUR_DOMAIN]/CN=Certificate Authority
1 s:/O=PLATFORM.{OUR_DOMAIN]/CN=Certificate Authority
i:/O=PLATFORM.{OUR_DOMAIN]/CN=Certificate Authority
---
Server certificate
-----BEGIN CERTIFICATE-----

...here goes our certificate

-----END CERTIFICATE-----
subject=/O=PLATFORM.{OUR_DOMAIN]/CN={HOSTNAME}
issuer=/O=PLATFORM.{OUR_DOMAIN]/CN=Certificate Authority
---
No client certificate CA names sent
Server Temp Key: ECDH, ___, 521 bits
---
SSL handshake has read 2508 bytes and written 511 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol : TLSv1.2
    Cipher    : ___
    Session-ID: ___
    Session-ID-ctx:
    Master-Key: ___
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1429155044
    Timeout   : 300 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
---

Re: SSL handshake error when configuring SSL for Cloudera Navigator (cloudera-navigator.properties)

nataliaking_csc — Thu, 16 Apr 2015 04:03:18 GMT

I was looking some more to confirm that the issue is between Cloudera Navigator host and Cloudera Manager host:

2015-04-15 23:20:50,677 WARN 236787520@scm-web-23643:org.mortbay.log: SSL renegotiate denied: java.nio.channels.SocketChannel[connected local=/{CM_SERVER_HOST}:7183 remote=/{NAVIGATOR_HOST}:50359]

2015-04-15 23:20:57,174 WARN 236787520@scm-web-23643:org.mortbay.log: javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

Re: SSL handshake error when configuring SSL for Cloudera Navigator (cloudera-navigator.properties)

Grizzly — Thu, 16 Apr 2015 04:54:05 GMT

Have you established implicit trust for your private CA in the JDK layer as discussed in our documentation on encryption for the platform, as laid out here:

http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_sg_create_key_trust.html

You can import the CA's certificate into the [JAVA_HOME]/jre/lib/security path, using cacerts or jssecacerts as you see fit.

Trust in the JDK is documented in detail here as well

http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html#X509TrustManager

Re: SSL handshake error when configuring SSL for Cloudera Navigator (cloudera-navigator.properties)

nataliaking_csc — Thu, 16 Apr 2015 05:45:31 GMT

So far it appears that only Navigator is unhappy with the keystore. So I believe TLS/SSL was set up correctly, otherwise. We are using FreeIPA as certificate authority and below is a quick overview of steps taken to set it up, since there is a somewhat deviation from standard protocol.

On the Namenode (same host the Cloudera Manager lives on), I generated a certificate and key to be used by Cloudera Manager

# kinit -kt /etc/krb5.keytab

# ipa-getcert request -f cmhost.pem -k cmhost.key -r

# chmod 600 cmhost*

Then I copied the newly created cm-keys directory to each host.

$ for x in {LIST_OF_CDH_HOSTS}; do scp -r cm-keys $x:; done

$ for x in {LIST_OF_CDH_HOSTS}; do ssh -tty $x sudo bash -c "'mkdir -p /opt/cloudera/security/x509; mv cm-keys/* /opt/cloudera/security/x509; chown cloudera /opt/cloudera/security/x509/*'"; done

Next, I set up Puppet to configure Cloudera to use TLS.

Re: SSL handshake error when configuring SSL for Cloudera Navigator (cloudera-navigator.properties)

Grizzly — Thu, 16 Apr 2015 14:58:59 GMT

Ok so you have 2 paradigims that you are configuring for, the JDK, and OpenSSL

"Trust" is established differently between the two implementations, Navigator, being Java based, will derive trust through the default JDK mechanisims I pointed out

For the Agents, Hue, and Implala the implementation is based on OpenSSL. Trust can be established for a root CA by directly configuring path to a CA pem file, or if there is a complex intermediary chain through providing the path to where the files are and running the c_rehash tool to generate the necessary base64 encoded symlinks.

Navigator is expecting you to configure a jks with private key and installed cert, but then as navigator attempts to connect to other cluster services that are using SSL/TLS, it needs to trust the CA that issued the server cert...

Re: SSL handshake error when configuring SSL for Cloudera Navigator (cloudera-navigator.properties)

nataliaking_csc — Thu, 16 Apr 2015 19:23:12 GMT

"Trust" is established differently between the two implementations, Navigator, being Java based, will derive trust through the default JDK mechanisms I pointed out

We are looking into whether we need to use keytool utility to generate those or use our FreeIPA server to generate certs for Navigator...

Re: SSL handshake error when configuring SSL for Cloudera Navigator (cloudera-navigator.properties)

Grizzly — Thu, 16 Apr 2015 20:48:05 GMT

And we cover import/export scenarios between the two implementations here:

http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_sg_openssl_jks.html

So you need to consider that you have a private key component, and a issued certificate component... the combination of the two in a JKS (private key and certificate) allow you to start up a java based service that will support SSL/TLS connections.

You can use your IPA based appoach as long as you have access to the private key and certificate it issues, to combine them and then import into a JKS.

The the CA certificate that issued the certificate, imported into the truststores I've discussed already, establishes inherent trust within java services, for SSL certificates created by that CA.

Re: SSL handshake error when configuring SSL for Cloudera Navigator (cloudera-navigator.properties)

nataliaking_csc — Thu, 16 Apr 2015 22:40:09 GMT

The the CA certificate that issued the certificate, imported into the truststores I've discussed already, establishes inherent trust within java services, for SSL certificates created by that CA.

We have already created a truststore on Namenode (node where Cloudera Manager is installed), when TLS was set up for all agents, that's what was used - /etc/cloudera-scm-server/keystore - and this file was copied to all nodes in Cloudera Hadoop cluster, including Navigator.

Re: SSL handshake error when configuring SSL for Cloudera Navigator (cloudera-navigator.properties)

Grizzly — Thu, 16 Apr 2015 23:02:35 GMT

Sorry,

So the term "truststore" is over-loaded. Are you saying a JKS file that you configure from the CM UI as a truststore for each service in the cluster, including management services? Or a "default" truststore like [JAVA_HOME]/jre/lib/security/cacerts (or jssecacerts) that establishes inherent trust as we have been discussing...

You have a number of configuration changes to make, one for each service, to recognize that trust store file (as opposed to instrumenting the JDK for trust).

Re: SSL handshake error when configuring SSL for Cloudera Navigator (cloudera-navigator.properties)

nataliaking_csc — Thu, 18 Jun 2015 15:34:32 GMT

Thank you! We got this to work finally. Now just need to wire to LDAP.

It looks like the keystore passed by following the instructions on Cloudera's site wasn't used for some reason. Following instructions here: http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_sg_create_key_trust.html#concept_u35_w2m_l4_unique_1

The upshot is that the keystore at /usr/java/jdk1.7.0_67-cloudera/jre/lib/security/jssecacerts (or ${JAVA_HOME}/lib/security/jssecacerts, where $JAVA_HOME is the home of the version of Java used by Cloudera Navigator, we used "ps" to find out where) should contain the root certificate. We then restarted both Cloudera Navigator services, and able to navigate to https://

Integration with FreeIPA was sort of confusing initially. Thanks for your help in understanding the mechnisms of this functionality.