After enabling TLS cloudera agent heartbeat failing

[#1]

Version: Cloudera Express 5.15.0 

Java VM Name: Java HotSpot(TM) 64-Bit Server VM

Java VM Vendor: Oracle Corporation

Java Version: 1.7.0_67

System details:

Linux optim-rhel72-uppu.development.unicomglobal.software 3.10.0-327.28.3.el7.x86_64 #1 SMP Fri Aug 12 13:21:05 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux

It is a single host and I am using self signed certificate. I am just validating a POC with one of my product and hence not yet licensed.

Followed the steps mentioned at this link: 

https://www.cloudera.com/documentation/enterprise/5-11-x/topics/how_to_configure_cm_tls.html

https://www.cloudera.com/documentation/enterprise/5-15-x/topics/sg_self_signed_tls.html

After enabling TLS, cloudera agant heartbeat is failing with the below lines in the cloudera-scm-agent.log

[27/Dec/2018 20:58:28 +0000] 6869 MainThread agent        ERROR    Heartbeating to optim-rhel72-uppu.development.unicomglobal.software:7182 failed.
Traceback (most recent call last):
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.15.0-py2.7.egg/cmf/agent.py", line 1424, in _send_heartbeat
    self.max_cert_depth)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.15.0-py2.7.egg/cmf/https.py", line 138, in __init__
    self.conn.connect()
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/M2Crypto-0.24.0-py2.7-linux-x86_64.egg/M2Crypto/httpslib.py", line 59, in connect
    sock.connect((self.host, self.port))
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/M2Crypto-0.24.0-py2.7-linux-x86_64.egg/M2Crypto/SSL/Connection.py", line 195, in connect
    ret = self.connect_ssl()
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/M2Crypto-0.24.0-py2.7-linux-x86_64.egg/M2Crypto/SSL/Connection.py", line 188, in connect_ssl
    return m2.ssl_connect(self.ssl, self._timeout)
SSLError: unexpected eof

Below lines in the cloudera-scm-server.log

2018-12-27 20:58:13,025 WARN 1320793343@agentServer-16:org.mortbay.log: javax.net.ssl.SSLHandshakeException: null cert chain
2018-12-27 20:58:28,034 WARN 1320793343@agentServer-16:org.mortbay.log: javax.net.ssl.SSLHandshakeException: null cert chain
2018-12-27 20:58:43,447 WARN 1320793343@agentServer-16:org.mortbay.log: javax.net.ssl.SSLHandshakeException: null cert chain
2018-12-27 20:58:58,082 WARN 1320793343@agentServer-16:org.mortbay.log: javax.net.ssl.SSLHandshakeException: null cert chain
2018-12-27 20:59:13,140 WARN 1320793343@agentServer-16:org.mortbay.log: javax.net.ssl.SSLHandshakeException: null cert chain

I have tried multiple times but none of them working. 

I didn't find any error while running this command:

openssl s_client -showcerts -connect optim-rhel72-uppu.development.unicomglobal.software:7182

Any help would be highly appreciated.

Thanks,

Tulasi

[#2]

Hi @Tulasi,

The exception in the agent and error in CM indicate that the agent is not presenting its certificate for authentication properly.  This is often due to something being wrong regarding your client_key_file configuration in /etc/cloudera-scm-agent/config.ini.

Can you check to see what is configured with and then verify the contents and that it is a valid PEM formatted key?

You can try this on the agent host to verify your agent has the right key and key password configured:

# openssl rsa -in `grep client_key_file /etc/cloudera-scm-agent/config.ini | sed 's/client_key_file=//'` -check -passin pass:`grep client_keypw_file /etc/cloudera-scm-agent/config.ini |sed 's/client_keypw_file=//'| xargs cat`

The output should show "RSA key ok" and the base64 encoded key text.

[#3]

Hi Bgooley,

Appreciate your quick response.

I have tried what you have suggested and the output also matching, see below:

[root@optim-rhel72-uppu security]# openssl rsa -in `grep client_key_file /etc/cloudera-scm-agent/config.ini | sed 's/client_key_file=//'` -check -passin pass:`grep client_keypw_file /etc/cloudera-scm-agent/config.ini |sed 's/client_keypw_file=//'| xargs cat`
RSA key ok
writing RSA key
-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEA5BoT3b00kTdMvhv9UO2Na3//1n+HNcD9nxH4ZVW/Ye2HeY+3

<<<< other lines>>>>

C5bNRh3+6bdksCaDFXB85cm96iZfmbZIO4oks5fomkuvBpPa3izjAQ==
-----END RSA PRIVATE KEY-----

[root@optim-rhel72-uppu security]#

Let me know if you need anything else.

Thanks,

Tulasi

[#4]

This shows the client_key_file settings are fine, please also verify that the CM agent can access the files referenced as client_key_file and client_key_file in /etc/cloudera-scm-agent/config.ini, e.g. with command 

namei -l <path-to-file-name>

[#5]

This is what I see:

[root@optim-rhel72-uppu ~]# namei -l /opt/cloudera/security/pki
f: /opt/cloudera/security/pki
dr-xr-xr-x root         root         /
drwxr-xr-x root         root         opt
drwxr-xr-x cloudera-scm cloudera-scm cloudera
drwxr-xr-x cloudera-scm cloudera-scm security
drwxr-xr-x cloudera-scm cloudera-scm pki

[root@optim-rhel72-uppu ~]# namei -l  /etc/cloudera-scm-agent
f: /etc/cloudera-scm-agent
dr-xr-xr-x root root /
drwxr-xr-x root root etc
drwxr-xr-x root root cloudera-scm-agent

[#6]

Sorry for being unclear but the command needs to have the full path including the actual filename as specified in config.ini for client_key_file and client_keypw_file

[#7]

Thanks for looking at it, here is the output and let me know if anything else is required.

[root@optim-rhel72-uppu ~]# namei -l /opt/cloudera/security/pki/agent.key
f: /opt/cloudera/security/pki/agent.key
dr-xr-xr-x root         root         /
drwxr-xr-x root         root         opt
drwxr-xr-x cloudera-scm cloudera-scm cloudera
drwxr-xr-x cloudera-scm cloudera-scm security
drwxr-xr-x cloudera-scm cloudera-scm pki
lrwxrwxrwx cloudera-scm cloudera-scm agent.key -> optim-rhel72-uppu.key
-rw-r--r-- cloudera-scm cloudera-scm   optim-rhel72-uppu.key
[root@optim-rhel72-uppu ~]# namei -l /etc/cloudera-scm-agent/agentkey.pw
f: /etc/cloudera-scm-agent/agentkey.pw
dr-xr-xr-x root root /
drwxr-xr-x root root etc
drwxr-xr-x root root cloudera-scm-agent
-rw-r--r-- root root agentkey.pw

[#8]

It is still not solved, any inputs would be appreciated.

Thanks,

Tulasi

[#9]

Hi,

We are sorry you are still encountering this issue. Since we know that the problem is isolated to TLS and that the agent is reporting a null certificate chain you will need to isolate why the certificate chain is null.

1.) Ensure that the certificates are in a standard x509 format for the agent.

2.) Ensure that the truststores/keystores on the CM host are in JCEKS format and not pkcs12.

3.) Make sure that the cloudera-scm user can read the Private Key, Certificates, Truststores, and Password Files.

4.) Make sure that the certificate on the failing agent contains a proper CN and DNS Alt Name if Alt Names are in use.

5.) Are you using self-signed certificates or certificates signed by a CA?

6.) If all else fails you can obtain a tcpdump of attempted communication with the server. The port that we normally heartbeat to is 7182. You can then review the conversation between the server and agent to attempt to identify at what point the error is returned and potentially what error is being observed at the protocol level. You can identify and restrict your tcpdump information by tcp.stream.

[#10]

Hi, Thanks for the response, I see one difference (in point 2) between document what you have noted. Document says to use PKCS12 format but you are suggesting JCEKS format. I didn't try with JCEKS, could you please confirm which format is correct. 2.) Ensure that the truststores/keystores on the CM host are in JCEKS format and not pkcs12. Document link https://www.cloudera.com/documentation/enterprise/5-15-x/topics/how_to_configure_cm_tls.html Thanks, Tulasi