SSLError: certificate verify failed

[#1]

How do I enable further debugging on cloudera-scm-agents?

I'm working on deploying the cluster using self signed certificates but I'm running into the below issue and can't get past it:

[07/Jul/2019 23:35:05 +0000] 23766 MainThread agent ERROR Heartbeating to cm-r01nn01.mws.mds.xyz:7182 failed.
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeat
self.cfg.max_cert_depth)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__
self.conn.connect()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect
sock.connect((self.host, self.port))
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connect
ret = self.connect_ssl()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_ssl
return m2.ssl_connect(self.ssl, self._timeout)
SSLError: certificate verify failed

What I have in my certificates folder is the following:

[root@cm-r01en01 pki]# pwd
/opt/cloudera/security/pki
[root@cm-r01en01 pki]# ls -atlri
total 16
69943167 -rw-r--r--. 1 root root 2385 Apr  1 23:06 cm-r01en01.mws.mds.xyz.keystore.jks
69943152 -rw-r--r--. 1 root root 1453 Apr  1 23:07 cm-r01en01.mws.mds.xyz.pem
 3870062 drwxr-xr-x. 5 root root   37 Apr  1 23:09 ..
69943169 lrwxrwxrwx. 1 root root   62 Apr  1 23:11 server.jks -> /opt/cloudera/security/pki/cm-r01en01.mws.mds.xyz.keystore.jks
69943259 -rw-r--r--. 1 root root 1453 Jul  6 20:01 cm-r01nn01.mws.mds.xyz.pem
69943154 lrwxrwxrwx. 1 root root   53 Jul  6 20:02 rootca.pem -> /opt/cloudera/security/pki/cm-r01nn01.mws.mds.xyz.pem
67689060 lrwxrwxrwx. 1 root root   53 Jul  6 20:36 agent.pem -> /opt/cloudera/security/pki/cm-r01en01.mws.mds.xyz.pem
69943151 drwxr-xr-x. 2 root root 4096 Jul  6 20:36 .
[root@cm-r01en01 pki]#

I'm not 100% sure if I have everything right though.  My cloudera-scm-agent config for that one host:

[root@cm-r01en01 pki]# cat /etc/cloudera-scm-agent/config.ini|grep -v "#" | sed -e "/^$/d"
[General]
server_host=cm-r01nn01.mws.mds.xyz
server_port=7182
max_collection_wait_seconds=10.0
metrics_url_timeout_seconds=30.0
task_metrics_timeout_seconds=5.0
monitored_nodev_filesystem_types=nfs,nfs4,tmpfs
local_filesystem_whitelist=ext2,ext3,ext4,xfs
impala_profile_bundle_max_bytes=1073741824
stacks_log_bundle_max_bytes=1073741824
stacks_log_max_uncompressed_file_size_bytes=5242880
orphan_process_dir_staleness_threshold=5184000
orphan_process_dir_refresh_interval=3600
scm_debug=DEBUG
dns_resolution_collection_interval_seconds=60
dns_resolution_collection_timeout_seconds=30
[Security]
use_tls=1
max_cert_depth=9
verify_cert_file=/opt/cloudera/security/pki/agent.pem
verify_cert_dir=/opt/cloudera/security/pki/
[Hadoop]
[Cloudera]
[JDBC]
[Cgroup_Paths]
[root@cm-r01en01 pki]#

cm-r01nn01 is the Name Node.

cm -r01en01 will be the gateway  / entry point to the cluster.  It will also run a few services.  

This is CM 6.2 .  I'm looking to go through the certificate process in preparation for a more formal deployment later on w/ official certificates.  Using self signed certs for now for this POC.

In particular, what certificate has it tried to load and is looking for?  How do I enable further debug logs to see all the calls it's making and files it's loading?

Cheers,
TK

[#2]

@TCloud,

Yeah, the deployment assumes that each HAProxy would be on its own host.  See the diagram here:

https://www.cloudera.com/documentation/enterprise/latest/topics/admin_cm_ha_hosts.html#concept_qkr_b...

and general steps:

https://www.cloudera.com/documentation/enterprise/latest/topics/admin_cm_ha_hosts.html#concept_uxy_5...

[#3]

So I've moved the HAproxy and Keepalived to another set of hosts.  Same issue with certs however I did try to replace the cm-r01nn01 and cm-r01nn02 (Name Node's) with the SAN SSL Certificate that I've generated.  Here is the writeup:

==> /var/log/cloudera-scm-agent/cloudera-scm-agent.log <==
self.cfg.max_cert_depth)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__
self.conn.connect()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect
sock.connect((self.host, self.port))
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connect
ret = self.connect_ssl()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_ssl
return m2.ssl_connect(self.ssl, self._timeout)
SSLError: certificate verify failed

Replace the certificate cm-r01nn01 and cm-r01nn02 with the SAN SSL Cert one as follows:

[root@cm-r01nn01 yum.repos.d]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit | grep -Ei "cm|srv"
cm-r01wn07.mws.mds.xyz, Jul 13, 2019, PrivateKeyEntry,
cm-r01wn04.mws.mds.xyz, Jul 13, 2019, PrivateKeyEntry,
cm-r01wn01.mws.mds.xyz, Jul 13, 2019, PrivateKeyEntry,
cm-r01wn08.mws.mds.xyz, Jul 13, 2019, PrivateKeyEntry,
cm-r01nn01.mws.mds.xyz, Jul 12, 2019, PrivateKeyEntry,
cm-r01wn05.mws.mds.xyz, Jul 13, 2019, PrivateKeyEntry,
cm-r01wn02.mws.mds.xyz, Jul 13, 2019, PrivateKeyEntry,
cm-r01en01.mws.mds.xyz, Jul 12, 2019, PrivateKeyEntry,
acraizfnmt-rcm, Mar 26, 2019, trustedCertEntry,
cm-r01nn02.mws.mds.xyz, Jul 12, 2019, PrivateKeyEntry,
cm-r01wn06.mws.mds.xyz, Jul 13, 2019, PrivateKeyEntry,
cm-r01wn03.mws.mds.xyz, Jul 13, 2019, PrivateKeyEntry,
srv-c01.mws.mds.xyz, Jul 18, 2019, PrivateKeyEntry,
cm-r01en02.mws.mds.xyz, Jul 12, 2019, PrivateKeyEntry,
cm-c01.mws.mds.xyz, Jul 12, 2019, PrivateKeyEntry,

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12".
[root@cm-r01nn01 yum.repos.d]#
[root@cm-r01nn01 yum.repos.d]#
[root@cm-r01nn01 yum.repos.d]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit | grep -Ei "cm|srv"|grep -Ei r01nn01

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12".
cm-r01nn01.mws.mds.xyz, Jul 12, 2019, PrivateKeyEntry,
[root@cm-r01nn01 yum.repos.d]# keytool -delete -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit -alias cm-r01nn01.mws.mds.xyz ^C
[root@cm-r01nn01 yum.repos.d]#
[root@cm-r01nn01 yum.repos.d]#
[root@cm-r01nn01 yum.repos.d]#
[root@cm-r01nn01 yum.repos.d]# cd /root/srv-c01/
[root@cm-r01nn01 srv-c01]# ls -altri
total 40
201326721 dr-xr-x---. 11 root root 4096 Jul 17 22:04 ..
134330464 drwxr-xr-x. 2 root root 4096 Jul 18 22:39 1
2230259 -rw-r--r--. 1 root root 2422 Jul 18 22:46 srv-c01.mws.mds.xyz.keystore.jks
2230300 -rw-r--r--. 1 root root 2801 Jul 18 22:49 srv-c01.mws.mds.xyz.keystore.p12
2230651 -rw-r--r--. 1 root root 1763 Jul 18 22:49 srv-c01.mws.mds.xyz.cert.pem
2230652 -rw-r--r--. 1 root root 1860 Jul 18 22:49 srv-c01.mws.mds.xyz.key.pem
2230653 -rw-r--r--. 1 root root 1505 Jul 18 22:51 srv-c01.mws.mds.xyz.pem
2230654 -rw-r--r--. 1 root root 1679 Jul 18 23:00 srv-c01.mws.mds.xyz.key.nopass.pem
2230257 drwxr-xr-x. 3 root root 4096 Jul 18 23:00 .
2230655 -rw-r--r--. 1 root root 3442 Jul 18 23:00 srv-c01.mws.mds.xyz-haproxy.pem
[root@cm-r01nn01 srv-c01]# history|grep srv-c01.mws.mds.xyz.pem|grep openssl|tail
876 openssl x509 -in ./srv-c01.mws.mds.xyz.pem -noout -text
1029 history|grep srv-c01.mws.mds.xyz.pem|grep openssl|tail
[root@cm-r01nn01 srv-c01]# openssl x509 -in ./srv-c01.mws.mds.xyz.pem -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1594172762 (0x5f05255a)
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=US, ST=California, L=Los Angeles, O=MDS, OU=MDS, CN=srv-c01.mws.mds.xyz
Validity
Not Before: Jul 19 02:46:18 2019 GMT
Not After : Jul 16 02:46:18 2029 GMT
Subject: C=US, ST=California, L=Los Angeles, O=MDS, OU=MDS, CN=srv-c01.mws.mds.xyz
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:c5:a9:00:83:12:9e:02:86:32:4e:2b:a7:c6:1a:
6b:9d:e3:56:00:53:22:01:d8:db:83:cd:14:79:6a:
85:27:20:f6:5d:86:0e:0b:af:df:46:dd:c3:23:72:
f0:bf:38:3e:cd:9f:92:e6:65:81:7b:26:32:50:fc:
81:0e:7b:dd:b4:61:6f:a7:56:ec:c8:fe:89:72:ec:
e5:e0:63:61:92:77:0b:36:41:98:93:14:6d:53:a0:
24:fb:fb:77:40:98:5b:2f:d2:3c:65:4f:8b:65:33:
e5:db:14:ce:01:d2:4f:9f:e4:c6:c8:35:50:09:a2:
f3:48:0a:ac:06:fd:66:42:30:10:a4:e7:fa:a8:2b:
0b:2b:ef:ce:83:82:4e:0d:86:34:ce:0c:8d:0c:a2:
f5:88:4d:38:9f:3b:dd:2e:6e:e3:8c:60:69:da:8d:
a4:d4:db:d5:cd:26:91:95:ca:a2:47:de:3c:f3:8f:
52:b8:e5:b0:09:26:af:77:fb:a3:5b:40:f6:e8:1b:
66:d7:b7:1b:da:2c:6c:34:99:76:de:c4:9b:80:69:
25:d5:12:2f:cb:9b:c5:d2:7e:15:a7:50:5f:54:5c:
9d:6b:8c:c0:9c:03:3f:96:f3:8a:2c:a6:05:ec:a4:
d3:83:84:61:13:da:57:6d:e8:8c:93:d9:40:38:24:
96:c9
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication, Code Signing, E-mail Protection, Time Stamping, OCSP Signing
X509v3 Subject Alternative Name:
DNS:srv-c01.mws.mds.xyz, DNS:cm-r01nn01.mws.mds.xyz, DNS:cm-r01nn02.mws.mds.xyz
X509v3 Subject Key Identifier:
F6:EA:97:6F:82:20:84:75:E9:63:71:2F:16:D6:41:8B:64:05:07:0D
Signature Algorithm: sha256WithRSAEncryption
4f:35:6d:18:dc:5c:4a:65:db:8c:62:75:0b:f8:da:2b:14:72:
22:f7:3a:ba:15:17:58:41:46:3b:6b:6e:40:db:6b:be:e5:07:
82:d1:37:0a:d6:4e:96:14:f6:87:ca:ff:d3:5f:a9:94:de:81:
e7:a1:28:94:0a:19:0b:f4:dc:ed:0a:a5:77:78:20:53:3f:3f:
03:54:67:a0:c4:a1:de:49:7d:e8:fc:2d:76:bd:7b:a5:98:cd:
45:7e:ba:21:79:e2:91:7d:f3:e9:d6:5d:b7:91:34:30:3a:e4:
3a:38:e9:33:9b:26:2e:3e:6c:c9:3d:5d:48:81:cb:35:2f:ff:
7a:ff:22:c2:f8:b5:a2:01:d0:54:7f:f2:08:33:89:78:80:af:
72:2d:d7:df:61:f0:4a:7f:d2:19:0d:c6:0c:51:ee:4e:c1:ed:
8d:8b:4f:82:17:47:6b:03:1a:f2:8b:00:cc:17:8a:75:ca:72:
c0:a4:a7:12:87:32:16:89:15:2c:80:d1:07:fd:37:e8:bf:f5:
87:6b:a2:dd:9d:a4:c4:2c:68:f8:d9:15:dd:3c:40:6d:8b:e0:
6d:c4:87:6d:39:a9:6b:91:f6:0a:bc:7c:63:e7:f0:37:cb:7a:
5f:35:6c:5c:f9:bb:cb:58:1a:b9:9c:49:ab:24:ac:2a:c9:2d:
3f:b2:2f:68
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit | grep -Ei "cm|srv"|grep -Ei r01nn01

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12".
cm-r01nn01.mws.mds.xyz, Jul 12, 2019, PrivateKeyEntry,
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit | grep -Ei "cm|srv"|grep -Ei r01nn01

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12".
cm-r01nn01.mws.mds.xyz, Jul 12, 2019, PrivateKeyEntry,
[root@cm-r01nn01 srv-c01]# keytool -delete -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit -alias cm-r01nn01.mws.mds.xyz

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12".
[root@cm-r01nn01 srv-c01]# keytool -delete -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit -alias cm-r01nn02.mws.mds.xyz

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12".
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]# keytool -importkeystore -srckeystore srv-c01.mws.mds.xyz.keystore.jks -destkeystore srv-c01.mws.mds.xyz.keystore.p12 -deststoretype PKCS12 -destalias cm-r01nn01.mws.mds.xyz -srcalias srv-c01.mws.mds.xyz -deststorepass srv-c01.mws.mds.xyz -destkeypass srv-c01.mws.mds.xyz
Importing keystore srv-c01.mws.mds.xyz.keystore.jks to srv-c01.mws.mds.xyz.keystore.p12...
Enter source keystore password:
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit | grep -Ei "cm|srv"|grep -Ei r01nn

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12".
[root@cm-r01nn01 srv-c01]# keytool -delete -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit -alias cm-r01nn01.mws.mds.xyz
keytool error: java.lang.Exception: Alias <cm-r01nn01.mws.mds.xyz> does not exist
[root@cm-r01nn01 srv-c01]# keytool -list -keystore srv-c01.mws.mds.xyz.keystore.p12
Enter keystore password:
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 2 entries

srv-c01.mws.mds.xyz, Jul 18, 2019, PrivateKeyEntry,
Certificate fingerprint (SHA1): 01:2D:6C:36:E6:7D:76:2B:45:66:91:7F:E8:B9:C2:61:A4:22:FB:D6
cm-r01nn01.mws.mds.xyz, Jul 26, 2019, PrivateKeyEntry,
Certificate fingerprint (SHA1): 01:2D:6C:36:E6:7D:76:2B:45:66:91:7F:E8:B9:C2:61:A4:22:FB:D6
[root@cm-r01nn01 srv-c01]# keytool -delete -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit -alias cm-r01nn01.mws.mds.xyz
keytool error: java.lang.Exception: Alias <cm-r01nn01.mws.mds.xyz> does not exist
[root@cm-r01nn01 srv-c01]# keytool -delete -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit -destalias cm-r01nn01.mws.mds.xyz
Enter alias name: cm-r01nn01.mws.mds.xyz
keytool error: java.lang.Exception: Alias <cm-r01nn01.mws.mds.xyz> does not exist
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]# keytool -delete -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass cm-r01nn01.mws.mds.xyz -alias cm-r01nn01.mws.mds.xyz
keytool error: java.io.IOException: Keystore was tampered with, or password was incorrect
[root@cm-r01nn01 srv-c01]# keytool -delete -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass srv-c01.mws.mds.xyz -alias cm-r01nn01.mws.mds.xyz
keytool error: java.io.IOException: Keystore was tampered with, or password was incorrect
[root@cm-r01nn01 srv-c01]# keytool -delete -keystore srv-c01.mws.mds.xyz.keystore.p12 -storepass srv-c01.mws.mds.xyz -alias cm-r01nn01.mws.mds.xyz [root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]# keytool -list -keystore srv-c01.mws.mds.xyz.keystore.p12 Enter keystore password:
keytool error: java.io.IOException: keystore password was incorrect
[root@cm-r01nn01 srv-c01]# keytool -list -keystore srv-c01.mws.mds.xyz.keystore.p12
Enter keystore password:
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 1 entry

srv-c01.mws.mds.xyz, Jul 18, 2019, PrivateKeyEntry,
Certificate fingerprint (SHA1): 01:2D:6C:36:E6:7D:76:2B:45:66:91:7F:E8:B9:C2:61:A4:22:FB:D6
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]# history|grep import|grep srv-c01.mws.mds.xyz.keystore.p12
699 keytool -importkeystore -srckeystore srv-c01.mws.mds.xyz.keystore.jks -destkeystore srv-c01.mws.mds.xyz.keystore.p12 -deststoretype PKCS12 -srcalias srv-c01.mws.mds.xyz -deststorepass srv-c01.mws.mds.xyz
869 keytool -importkeystore -srckeystore srv-c01.mws.mds.xyz.keystore.jks -destkeystore srv-c01.mws.mds.xyz.keystore.p12 -deststoretype PKCS12 -srcalias srv-c01.mws.mds.xyz -deststorepass srv-c01.mws.mds.xyz -destkeypass srv-c01.mws.mds.xyz
1034 keytool -importkeystore -srckeystore srv-c01.mws.mds.xyz.keystore.jks -destkeystore srv-c01.mws.mds.xyz.keystore.p12 -deststoretype PKCS12 -destalias cm-r01nn01.mws.mds.xyz -srcalias srv-c01.mws.mds.xyz -deststorepass srv-c01.mws.mds.xyz -destkeypass srv-c01.mws.mds.xyz
1044 history|grep import|grep srv-c01.mws.mds.xyz.keystore.p12
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]# keytool -importkeystore -srckeystore srv-c01.mws.mds.xyz.keystore.jks -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -srcalias srv-c01.mws.mds.xyz -deststorepass changeit -srcstorepass srv-c01.mws.mds.xyz -destalias cm-r01nn01.mws.mds.xyz
Importing keystore srv-c01.mws.mds.xyz.keystore.jks to /etc/pki/ca-trust/extracted/java/jssecacerts...

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12".
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]# keytool -importkeystore -srckeystore srv-c01.mws.mds.xyz.keystore.jks -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -srcalias srv-c01.mws.mds.xyz -deststorepass changeit -srcstorepass srv-c01.mws.mds.xyz -destalias cm-r01nn02.mws.mds.xyz
Importing keystore srv-c01.mws.mds.xyz.keystore.jks to /etc/pki/ca-trust/extracted/java/jssecacerts...

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12".
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#





[root@cm-r01nn01 srv-c01]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit -alias srv-c01.mws.mds.xyz -alias cm-r01nn01.mws.mds.xyz -v
Alias name: cm-r01nn01.mws.mds.xyz
Creation date: Jul 26, 2019
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=srv-c01.mws.mds.xyz, OU=MDS, O=MDS, L=Los Angeles, ST=California, C=US
Issuer: CN=srv-c01.mws.mds.xyz, OU=MDS, O=MDS, L=Los Angeles, ST=California, C=US
Serial number: 5f05255a
Valid from: Thu Jul 18 22:46:18 EDT 2019 until: Sun Jul 15 22:46:18 EDT 2029
Certificate fingerprints:
MD5: A7:C0:9E:E4:CC:DC:7E:4B:3A:96:CF:11:58:6C:86:D2
SHA1: 01:2D:6C:36:E6:7D:76:2B:45:66:91:7F:E8:B9:C2:61:A4:22:FB:D6
SHA256: 62:11:52:4E:7C:1E:03:11:20:CA:41:7E:5D:22:64:F8:CE:CC:85:C6:07:06:A9:21:FE:25:F3:71:DD:20:00:49
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
serverAuth
clientAuth
codeSigning
emailProtection
timeStamping
OCSPSigning
]

#2: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
DNSName: srv-c01.mws.mds.xyz
DNSName: cm-r01nn01.mws.mds.xyz
DNSName: cm-r01nn02.mws.mds.xyz
]

#3: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: F6 EA 97 6F 82 20 84 75 E9 63 71 2F 16 D6 41 8B ...o. .u.cq/..A.
0010: 64 05 07 0D d...
]
]


Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12".
[root@cm-r01nn01 srv-c01]#







[root@cm-r01wn08 srv-c01]#
[root@cm-r01wn08 srv-c01]# keytool -importkeystore -srckeystore srv-c01.mws.mds.xyz.keystore.jks -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -srcalias srv-c01.mws.mds.xyz -deststorepass changeit -srcstorepass srv-c01.mws.mds.xyz -destalias cm-r01nn01.mws.mds.xyz
Importing keystore srv-c01.mws.mds.xyz.keystore.jks to /etc/pki/ca-trust/extracted/java/jssecacerts...

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12".
[root@cm-r01wn08 srv-c01]# keytool -importkeystore -srckeystore srv-c01.mws.mds.xyz.keystore.jks -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -srcalias srv-c01.mws.mds.xyz -deststorepass changeit -srcstorepass srv-c01.mws.mds.xyz -destalias cm-r01nn02.mws.mds.xyz
Importing keystore srv-c01.mws.mds.xyz.keystore.jks to /etc/pki/ca-trust/extracted/java/jssecacerts...

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12".
[root@cm-r01wn08 srv-c01]#
[root@cm-r01wn08 srv-c01]#
[root@cm-r01wn08 srv-c01]#
[root@cm-r01wn08 srv-c01]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit | grep -Ei "cm|srv" srv-c01.mws.mds.xyz, Jul 21, 2019, PrivateKeyEntry,
acraizfnmt-rcm, Mar 26, 2019, trustedCertEntry,
cm-r01wn08.mws.mds.xyz, Jul 10, 2019, trustedCertEntry,
cm-r01nn02.mws.mds.xyz, Jul 26, 2019, PrivateKeyEntry,
cm-r01nn01.mws.mds.xyz, Jul 26, 2019, PrivateKeyEntry,

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12".
[root@cm-r01wn08 srv-c01]#
[root@cm-r01wn08 srv-c01]#

But no luck.  Again, is there any way to increase the debug level on the cloudera-scm-agent so it will print further information detailing what private and public key files it's opening up?

Thx,
TK

[#4 · Accepted solution]

@TCloud,

The exception is in the agent and indicates to us that the agent is not able to verify the certificate that was returned by Cloudera Manager during the TLS handshake.

In order to know why, we should look at what host the agent tried to contact (server_host in config.ini) and what certificates were listed in the SAN of the server certificate.

You can use the following command to see what certificate is returned:

openssl s_client -connect $(grep "server_host" /etc/cloudera-scm-agent/config.ini | sed s/server_host=//):7182 </dev/null | openssl x509 -text -noout

Then, check to make sure agent's truststore has the proper certificate that trusts the CM cert.  To test, you can use:

openssl s_client -connect $(grep -v '^#' /etc/cloudera-scm-agent/config.ini | grep "server_host=" | sed s/server_host=//):7182 -CAfile $(grep -v '^#' /etc/cloudera-scm-agent/config.ini | grep "verify_cert_file=" |sed s/verify_cert_file=//) -verify_hostname $(grep -v '^#' /etc/cloudera-scm-agent/config.ini | grep "server_host=" | sed s/server_host=//)</dev/null

The above is probably not that elegant, but you should be able to run it as it is.  It will grab your hostname and trust store file from the host's config.ini and then connect to your CM host to do a TLS handshake.  "-verify_hostname" will tell openssl to also do hostname validation to mimic what the agent does.

The result code of the above command should give us a better idea of why the handshake is failing.

[#5]

Realizing I didn't close this off.  

The suggestions in this post worked perfectly to move me along and eventually setup full TLS encryption.

Thanks very much guy's for the help.  Very much appreciated!