Cloudera SCM Agent SSL SSL: CERTIFICATE_VERIFY_FAILED

[#1]

We are using CDH 5.8.3 and agent error has SSL: CERTIFICATE_VERIFY_FAILED even the verify cert dir location has specifiied. Below is the error we're seeing in logs :

Root ca has been copied to the /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem as instructed in one of the thread along with jsse certs.

<<<hostname is valid and we see correct one in logs>>>

[05/Sep/2018 21:53:39 +0000] 31359 Thread-13 https ERROR Failed to retrieve/stroe URL: https://<hostname>:7183/cmf/parcel/download/CDH-5.8.3-1.cdh5.8.3.p0.2-el7.parcel.torrent -> /opt/cloudera/parcel-cache/CDH-5.8.3-1.cdh5.8.3.p0.2-el7.parcel.torrent <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)>
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.3-py2.7.egg/cmf/https.py", line 175, in fetch_to_file
resp = self.open(req_url)
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.3-py2.7.egg/cmf/https.py", line 170, in open
return self.opener(*pargs, **kwargs)
File "/usr/lib64/python2.7/urllib2.py", line 431, in open
response = self._open(req, data)
File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
'_open', req)
File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/usr/lib64/python2.7/urllib2.py", line 1258, in https_open
context=self._context, check_hostname=self._check_hostname)
File "/usr/lib64/python2.7/urllib2.py", line 1214, in do_open
raise URLError(err)
URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)>

Please help.

[#2]

Without knowing more details it looks like your agent does not have (configured or stored) the certificate of the Cloudera Manager server, thus cannot connect to 7183 port. Make sure you follow Cloudera's guide how to setup TLS
https://www.cloudera.com/documentation/enterprise/5-8-x/topics/cm_sg_config_tls_security.html

[#3]

On CM host - Keystore file has been generated and root ca is imported to the same keystore file. In the newly added host, i have copied the cert files under /opt/cloudera/security/pki/ , jsse file and updated the same value in config.ini as indicated in the documentation. I'm not if I missed anything. Any ideas would be really helpful.

Thanks.

[#4]

I did perform rehash after installing open-ssl-perl package. Still we're see SSLError: certificate verify failed on agent logs. 

[root@hostname pki]# ls -tlr
total 16
-rw-r--r-- 1 root root 1834 Sep 5 20:54 ca-key
-rw-r--r-- 1 root root 1314 Sep 5 20:59 ca-cert
lrwxrwxrwx 1 root root 7 Sep 6 11:58 4ba83bb9.0 -> ca-cert

Looks like no where to go further :(. Ideas please ...

[#5]

@Krish216,

The solution has been applied many times, so it is most likely that a minor mistake was made.  How did you add the CM certificate to the /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem file?

What exactly did you add?  How did you get that text?

PEM files can be very sensitive to missing "-" in the headers of certificates.

also, it could be possible you copied/pasted and some extra characters got in there...

Try running:

openssl s_client -connect <cm_host>:7183 -CAfile /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem < /dev/null

If openssl connects without a problem, your agent on the same host should too.

Make sure you have restarted the agent after making the pem file update:

# service cloudera-scm-agent restart

[#6]

Hi @bgooley, Thank you for the response.

Here are the contents - On CM host below CA cert is generated

[root@hostname pki]# ls -ltr
total 20
-rw-r--r-- 1 root root 1121 Sep 2 12:27 hostname-server.csr
-rw-r--r-- 1 root root 1834 Sep 2 12:27 ca-key
-rw-r--r-- 1 root root 1314 Sep 2 12:27 ca-cert
-rw-r--r-- 1 root root 4198 Sep 2 12:28 hostname.jks
lrwxrwxrwx 1 root root 7 Sep 6 10:21 4ba83cc1.0 -> ca-cert
[root@hostname pki]# pwd
/opt/cloudera/security/pki

-> ca cert is imported to keystore along with jsse certs on CM host and same copied to one of the agent hosts.

-> copied the contents for ca-cert generate through - openssl req -new -x509 -keyout ca-key -out ca-cert -days 3650

-> created the sym link for the ca-cert, updated the parameters in /etc/cloudera-scm-agent/config.ini along verify_cert_dir

-> Here is the output from verification -

[root@hostname1.domain cloudera-scm-agent]# openssl s_client -connect hostname:7183 -CAfile /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem < /dev/null
CONNECTED(00000003)
depth=0 C = Unknown, ST = Unknown, L = Unknown, O = Unknown, OU = Unknown, CN = hostname
verify return:1
---
Certificate chain
0 s:/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=hostname
i:/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=hostname
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDmTCCAoGgAwIBAgIEFdAE2jANBgkqhkiG9w0BAQsFADB9MRAwDgYDVQQGEwdV
bmtub3duMRAwDgYDVQQIEwdVbmtub3duMRAwDgYDVQQHEwdVbmtub3duMRAwDgYD
VQQKEwdVbmtub3duMRAwDgYDVQQLEwdVbmtub3duMSEwHwYDVQQDExhjc2VraGFy
MDIucWU5LmppZ3Nhdy5jb20wHhcNMTgwOTAyMTkyNzAyWhcNMjgwODMwMTkyNzAy
WjB9MRAwDgYDVQQGEwdVbmtub3duMRAwDgYDVQQIEwdVbmtub3duMRAwDgYDVQQH
EwdVbmtub3duMRAwDgYDVQQKEwdVbmtub3duMRAwDgYDVQQLEwdVbmtub3duMSEw
HwYDVQQDExhjc2VraGFyMDIucWU5LmppZ3Nhdy5jb20wggEiMA0GCSqGSIb3DQEB
AQUAA4IBDwAwggEKAoIBAQCg4ailB41WoGo6h3fepc6wILw37WaRZ3IApxj0ZTO0
n8po151baw0IEk1X+lYXAkuXloycCKsFUhjiOU2AF2iBbVOYXyA9IB5a9a+b507r
EG1NqdJrdN6k4kZrsv+91CizYunjNNheDIgUv/gA7U/sSbvvchXUlN+j4y9LpwKH
wcbY/kKLZV9EoNeyrJANccHYiHRQgPLfEcQoHJtDRuWt0TIDqZOR+Xe6fICLfDiC
8WRXqpYKRfFoLVwzFEJuilZXT/J9obzhRgxt9sF7eVV86kFa2EIEZ7aAzZGdIeVT
s3RpUO/iSq2X5DxQ7BH8Duc6kgfuaAIOCuANeEGO7trxAgMBAAGjITAfMB0GA1Ud
DgQWBBTPmi/hBkF9xJxiN4bvB2QehVuM5TANBgkqhkiG9w0BAQsFAAOCAQEAJEgp
747BfFBIKVPoBCOnxopLM0iJ0jhOTMPo2gld8XoHdH52UOKPlnTsYxqRv9qcU2m8
zjlNxpzT2LMEMGWwk8cP+4ikGrW1vI/bUZOVwFg2pDeho/AE9IOgdn6kd9OQXnL+
D2AB8woJwLWEQ755CEETjvk84kC4BVMMCAB4tDhwAEmzhsehg5B1FzVGfNE5k7fS
Ey+ZkNMr/9omWYd4lzHja8pCBCISUCVU7c4ybnEdvLfhyGk/uoNzE2Tf11OsmNtj
Fqc4Wbfr9BmcEzDWP6pO5ID6hOu93JaKPmMiW6ofzoJfUdF9Mj/q8pEBc88dqpbf
aMvL2RAdfPcFFqSEWA==
-----END CERTIFICATE-----
subject=/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=hostname
issuer=/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=hostname
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1409 bytes and written 415 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 5B91D501E15D42E800371EBAFE7BF3FD673EEEDA1A10E30CBEC808C2431F2325
Session-ID-ctx:
Master-Key: A0DD11CA122343D962AFE236893EC2F00371D37DF3BDC340AB3F1CCDC25C8E48F4DC28255A6CC1926654D1708FE23B9A
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1536283905
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
DONE

restarted CM server and agent

Please advise if I'm missing anything.