Cloudera agent SSL error

[#1]

Hi team,

I am trying to enable SSL in transit for my cloudera cluster using the document https://www.cloudera.com/documentation/enterprise/latest/topics/how_to_configure_cm_tls.html

I am able to successfully configure till the step "Enable Server Certificate Verification on Cloudera Manager Agents" , however once i completed "Configure Agent Certificate Authentication" , i am receiving the below error and all the hosts are in bad health state, could you please help ?

[27/Mar/2019 11:12:37 +0000] 1022 MainThread agent ERROR Heartbeating to cmhost.antuit.internal:7182 failed.
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1388, in _send_heartbeat
self.cfg.max_cert_depth)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__
self.conn.connect()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 80, in connect
sock.connect((self.host, self.port))
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 305, in connect
ret = self.connect_ssl()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 292, in connect_ssl
return m2.ssl_connect(self.ssl, self._timeout)
SSLError: sslv3 alert certificate unknown

Notes:

-----

1. I am using a private intermediate CA to sign the certificates for each host.

2. I have imported both root and intermediate CA certs into jssecacerts in the cloudera manager host 

3. I am able to manually verify the signed certs

cat ca.cert.pem intermediate.cert.pem > verify.pem

sudo openssl verify -CAfile verifier.pem cmhost.XX.YY.pem
cmhost.XX.YY.pem: OK

Thanks,

Chiranjeevi

[#2]

Did you follow the steps as explained in Enable Server Certificate Verification on Cloudera Manager Agents ?

The agent configuration needs to be given a PEM file for the verify_cert_file configuration property which holds the root CA certificate.

To verify the configuration, please run this command on one of the cluster nodes, with <filename> being the full path to file as specified in verify_cert_file:

# openssl s_client -CAfile <filename> cmhost.antuit.internal:7182

[#3]

Hi,

Yes i have completed the steps in "Enable server certificate verification" . Please find details below, 

Note: The verifier.pem file has both RootCA and IntermediateCA certificates and cmhost.antuit.internal.pem has the signed certificate + IntermediateCA certificate

# A file of CA certificates in PEM format. The file can contain several CA
# certificates identified by
#
# -----BEGIN CERTIFICATE-----
# ... (CA certificate in base64 encoding) ...
# -----END CERTIFICATE-----
#
# sequences. Before, between, and after the certificates text is allowed which
# can be used e.g. for descriptions of the certificates.
#
# The file is loaded once, the first time an HTTPS connection is attempted. A
# restart of the agent is required to pick up changes to the file.
#
# Note that if neither verify_cert_file or verify_cert_dir is set, certificate
# verification will not be performed.
verify_cert_file=/opt/cloudera/security/pki/verifier.pem

[root@cmhost pki]# openssl s_client -connect cmhost.antuit.internal:7182 -CAfile verifier.pem -cert cmhost.antuit.internal.pem -key agent.key
Enter pass phrase for agent.key:
CONNECTED(00000003)
depth=2 C = IN, ST = KA, L = BNG, O = Antuit, OU = DE, CN = Antuit Root CA
verify return:1
depth=1 C = IN, ST = KA, O = Antuit, OU = DE, CN = Antuit Inter CA
verify return:1
depth=0 C = IN, ST = KA, L = BNG, O = Antuit, OU = DE, CN = cmhost.antuit.internal
verify return:1
140606215886736:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:s3_pkt.c:1493:SSL alert number 46
140606215886736:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:
---
Certificate chain
0 s:/C=IN/ST=KA/L=BNG/O=Antuit/OU=DE/CN=cmhost.antuit.internal
i:/C=IN/ST=KA/O=Antuit/OU=DE/CN=Antuit Inter CA
1 s:/C=IN/ST=KA/O=Antuit/OU=DE/CN=Antuit Inter CA
i:/C=IN/ST=KA/L=BNG/O=Antuit/OU=DE/CN=Antuit Root CA
---

[#4]

Please make the verifier.pem file contain only the root CA certificate. Then list contents with 

# openssl x509 -text -in /opt/cloudera/security/pki/verifier.pem

And repeat the connection test with this exact command

# openssl s_client -connect cmhost.antuit.internal:7182 -CAfile /opt/cloudera/security/pki/verifier.pem