Issue After enabling the TLS level 1 encryption

[#1]

Hi,

I have enabled the TLS level 1 encryption and after the same I am getting few errors in my log as per below;

1] Getting below error in My cloudera-scm-server.log

2017-08-16 14:56:56,261 INFO MainThread:com.cloudera.server.cmf.WebServerImpl: Cipher suite TLS_EMPTY_RENEGOTIATION_INFO_SCSV found. Allowing SSL/TLS renegotiations.
2017-08-16 14:56:56,288 INFO MainThread:com.cloudera.server.cmf.WebServerImpl: TLS web connections will use port: 7183
2017-08-16 14:56:56,292 INFO MainThread:com.cloudera.server.cmf.WebServerImpl: Plaintext web connections will use port: 7180
2017-08-16 14:56:56,337 INFO MainThread:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing command GenerateCredentials BasicCmdArgs{args=[]}.
2017-08-16 14:56:56,337 INFO MainThread:com.cloudera.server.cmf.Main: Generating credentials (command 4481) at startup
2017-08-16 14:56:56,393 INFO WebServerImpl:com.cloudera.enterprise.JavaMelodyFacade: No JavaMelody class net.bull.javamelody.SessionListener: net.bull.javamelody.SessionListener
2017-08-16 14:56:56,479 ERROR ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Unable to retrieve remote parcel repository manifest
java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused to http://serverip:8000/manifest.json
at com.ning.http.client.providers.netty.NettyResponseFuture.abort(NettyResponseFuture.java:297)
at com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:104)
at org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChannelFuture.java:399)
at org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChannelFuture.java:390)
at org.jboss.netty.channel.DefaultChannelFuture.setFailure(DefaultChannelFuture.java:352)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:409)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:366)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:282)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused to http://serverip:8000/manifest.json
at com.ning.http.client.providers.netty.NettyConnectListener.operationComplete(NettyConnectListener.java:100)
... 11 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.connect(NioClientSocketPipelineSink.java:404)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.processSelectedKeys(NioClientSocketPipelineSink.java:366)
at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink$Boss.run(NioClientSocketPipelineSink.java:282)
... 3 more

2017-08-16 15:31:35,624 INFO 1922557741@scm-web-39:com.cloudera.server.web.cmf.AuthenticationFailureEventListener: Authentication failure for user: '__cloudera_internal_user__mgmt-EVENTSERVER-bdec96eb8ea18d0be431197fa05f0a3b' from CMhost

2] Getting below error in my cloudera-scm-agent.log

ERROR    Heartbeating to CMhostname:7182 failed. Connection refused

Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.9.1-py2.6.egg/cmf/agent.py", line 1346, in _send_heartbeat
self.max_cert_depth)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.9.1-py2.6.egg/cmf/https.py", line 132, in __init__
self.conn.connect()
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/M2Crypto-0.21.1-py2.6-linux-x86_64.egg/M2Crypto/httpslib.py", line 50, in connect
self.sock.connect((self.host, self.port))
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/M2Crypto-0.21.1-py2.6-linux-x86_64.egg/M2Crypto/SSL/Connection.py", line 181, in connect
self.socket.connect(addr)
File "<string>", line 1, in connect
error: [Errno 111] Connection refused

ERROR    [1646-cloudera-mgmt-HOSTMONITOR] Failed to update

3] In Eventserver log file

2017-08-16 13:28:30,475 ERROR com.cloudera.cmf.eventcatcher.server.EventCatcherService: Error starting EventServer
org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@/var/lib/cloudera-scm-eventserver/v3/write.lock

Can anybody please help me on the same, as I am not able to find out the proper solution for the same.

Thank you in advance.

Thanks,

Amit

[#2]

Did you do these steps prior to Level 1?

https://www.cloudera.com/documentation/enterprise/5-8-x/topics/cm_sg_tls_browser.html#xd_583c10bfdbd...

Did you check that your keystore contains the CM certificate and has the correct hostname?
Is the keystore file readable by the CM process user?

[#3]

Thanks mbigelow,

Yes, I have added the public CA certificate to keystore and I have given the user cloudera-scm a full permission on the keystore files like cacerts,jsscacerts, pki folder, x509 folder and jks folder.

I have validated the certificate using commands;

openssl s_client -showcerts -connect hostname:443

And

keytool -list -v -keystore cacerts --alias 

I have also validated that in the cloudera agent process file

/var/run/cloudera-scm-agent/process/1653-cloudera-mgmt-SERVICEMONITOR/cmon.conf

I can see some of the ssl entries as per below;

<property>
<name>scm.server.url</name>
<value>https://hostname:7183</value>
</property>

<property>
<name>com.cloudera.enterprise.ssl.client.truststore.location</name>
<value>/usr/java/jdk1.7.0_67-cloudera/jre/lib/security/cacerts</value>
</property>
<property>
<name>com.cloudera.enterprise.ssl.client.truststore.password</name>
<value>changeit</value>
</property>

Regarding your point "correct hostname in certificate" do I need to verify anything else, apart from what I mentioned above.

Also, I would be really thankful if you can suggest, what else I can do to fix these errors.

Thanks,

Amit

[#4 · Accepted solution]

This issue resolved for me when I rebooted my CM machine.

Thanks,

Amit

[#5]

@AmitAdhau

Could you kindly help me out with steps to deploy tls using self-signed certificate.