Cloudera management service are not getting started after TLS Encryption

[#1]

I have issued a self signed certificate to my Cloudera Manager node. and copied over all the nodes in the cluster. 

I am following steps given on this page http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_sg_tls_browser.html

I have restarted cloudera manager server and agents on all the nodes. Cloudera manager UI redirected to https://ip:7183 as expected. However when I tried to restart Cloudera Management Service it is not getting restarted and throwing below error.

2015-02-12 03:19:05,632 INFO CMMetricsForwarder-0:com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder: (29 skipped) Failed to send metrics.
java.lang.reflect.UndeclaredThrowableException
at com.sun.proxy.$Proxy101.writeMetrics(Unknown Source)
at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendWithAvro(ClouderaManagerMetricsForwarder.java:287)
at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendMetrics(ClouderaManagerMetricsForwarder.java:274)
at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.run(ClouderaManagerMetricsForwarder.java:129)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.avro.AvroRemoteException: java.net.ConnectException: Connection refused
at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:88)
... 11 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
at sun.net.www.http.HttpClient.New(HttpClient.java:308)
at sun.net.www.http.HttpClient.New(HttpClient.java:326)
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:996)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:932)
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:850)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1091)
at org.apache.avro.ipc.HttpTransceiver.writeBuffers(HttpTransceiver.java:71)
at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:58)
at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:72)
at org.apache.avro.ipc.Requestor.request(Requestor.java:147)
at org.apache.avro.ipc.Requestor.request(Requestor.java:101)
at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:72)
... 11 more

Any help appreciated. Am I missing something?

[#2]

"Trust"  now becomes an issue, for your self signed certificate.  You need to either provide a 'truststore' to the management services which is a jks file that contains you server certificate (not the private key, the signed certificate) so it can reference that "truststore" and validate your certificate when connecting.

There is discussion online about using opensssl to create a root and intermediary (subordinate) CA as well.   Using a self signed private CA makes establishing trust easier for a set of certificates signed by that CA.  Trust is established for the issuing CA rather than having to extract and gather each individual server's self signed certificate to a truststore.  The CA certificate present in a truststore is valid as well.  Each service instance (in this case the Management Services) then need to have a truststore they are pointing to with password to establish trust.  This can get over-complicated to manage.  Working with the JDK's default trust is much easier to manage. 

You can also extend the default trust store for the JDK, itself.  This is a much easier approach to manage.  It still does require managing the cacerts file in the $JAVA_HOME/jre/lib/security/ path as you upgrade JDK's.   Our documentation discusses keystores and truststores in the sections around where you are at.  The cacerts file, or its alternative named jssecacerts will be automatically used by the JDK when attempting to resolve trust for a public issued certificate's CA, or if added, any private CA that should be trusted by default within the JDK.

In this case with your self signed certificate you would copy that (the certificate) into whichever type of truststore approach you decide to use. 

The trust behavior is documented for the JDK, here:

http://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html#X509TrustManag...

(you can change the "7" in the URL for the JDK "6" or JDK "8" as well, its pretty much the same content across the releases.

The section in our docs covering keystores and truststores is here. 

http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cm_sg_create_key_trust....

[#3]

Grizzly,

Is there a way to disable TLS from the commandline? I have this issue as well and the Cloudera Manager doesn't even start.

https://community.cloudera.com/t5/Cloudera-Manager-Installation/Cannot-log-in-to-Cloudera-Manager-no...

Any help is appreciated.

Jim