question Re: SSLError: certificate verify failed in Support Questions

SSLError: certificate verify failed

TCloud — Fri, 16 Sep 2022 14:29:32 GMT

How do I enable further debugging on cloudera-scm-agents?

I'm working on deploying the cluster using self signed certificates but I'm running into the below issue and can't get past it:

[07/Jul/2019 23:35:05 +0000] 23766 MainThread agent ERROR Heartbeating to cm-r01nn01.mws.mds.xyz:7182 failed.
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeat
self.cfg.max_cert_depth)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__
self.conn.connect()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect
sock.connect((self.host, self.port))
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connect
ret = self.connect_ssl()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_ssl
return m2.ssl_connect(self.ssl, self._timeout)
SSLError: certificate verify failed

What I have in my certificates folder is the following:

[root@cm-r01en01 pki]# pwd
/opt/cloudera/security/pki
[root@cm-r01en01 pki]# ls -atlri
total 16
69943167 -rw-r--r--. 1 root root 2385 Apr  1 23:06 cm-r01en01.mws.mds.xyz.keystore.jks
69943152 -rw-r--r--. 1 root root 1453 Apr  1 23:07 cm-r01en01.mws.mds.xyz.pem
 3870062 drwxr-xr-x. 5 root root   37 Apr  1 23:09 ..
69943169 lrwxrwxrwx. 1 root root   62 Apr  1 23:11 server.jks -> /opt/cloudera/security/pki/cm-r01en01.mws.mds.xyz.keystore.jks
69943259 -rw-r--r--. 1 root root 1453 Jul  6 20:01 cm-r01nn01.mws.mds.xyz.pem
69943154 lrwxrwxrwx. 1 root root   53 Jul  6 20:02 rootca.pem -> /opt/cloudera/security/pki/cm-r01nn01.mws.mds.xyz.pem
67689060 lrwxrwxrwx. 1 root root   53 Jul  6 20:36 agent.pem -> /opt/cloudera/security/pki/cm-r01en01.mws.mds.xyz.pem
69943151 drwxr-xr-x. 2 root root 4096 Jul  6 20:36 .
[root@cm-r01en01 pki]#

I'm not 100% sure if I have everything right though. My cloudera-scm-agent config for that one host:

[root@cm-r01en01 pki]# cat /etc/cloudera-scm-agent/config.ini|grep -v "#" | sed -e "/^$/d"
[General]
server_host=cm-r01nn01.mws.mds.xyz
server_port=7182
max_collection_wait_seconds=10.0
metrics_url_timeout_seconds=30.0
task_metrics_timeout_seconds=5.0
monitored_nodev_filesystem_types=nfs,nfs4,tmpfs
local_filesystem_whitelist=ext2,ext3,ext4,xfs
impala_profile_bundle_max_bytes=1073741824
stacks_log_bundle_max_bytes=1073741824
stacks_log_max_uncompressed_file_size_bytes=5242880
orphan_process_dir_staleness_threshold=5184000
orphan_process_dir_refresh_interval=3600
scm_debug=DEBUG
dns_resolution_collection_interval_seconds=60
dns_resolution_collection_timeout_seconds=30
[Security]
use_tls=1
max_cert_depth=9
verify_cert_file=/opt/cloudera/security/pki/agent.pem
verify_cert_dir=/opt/cloudera/security/pki/
[Hadoop]
[Cloudera]
[JDBC]
[Cgroup_Paths]
[root@cm-r01en01 pki]#

cm-r01nn01 is the Name Node.

cm -r01en01 will be the gateway / entry point to the cluster. It will also run a few services.

This is CM 6.2 . I'm looking to go through the certificate process in preparation for a more formal deployment later on w/ official certificates. Using self signed certs for now for this POC.

In particular, what certificate has it tried to load and is looking for? How do I enable further debug logs to see all the calls it's making and files it's loading?

Cheers,
TK

Re: SSLError: certificate verify failed

bgooley — Mon, 08 Jul 2019 16:25:34 GMT

Hi @TCloud ,

Thank you for including all this information.

I see that you have both verify_cert_dir and verify_cert_file configured, but one or the other should be chosen. It appears you are configured for verify_cert_dir based on your directory listing.

I would recommend:

- commenting out verify_cert_dir by insertinga "#" character at the beginning of the line

- restarting the agent by executing "service cloudera-scm-server restart"

If that doesn't work, we can take a closer look at the certificates returned by Cloudera Manager and the certificates in your agent.pem file.

Re: SSLError: certificate verify failed

TCloud — Tue, 09 Jul 2019 04:31:16 GMT

First, appreciate you looking into this.

Just left verify_cert_file in place in /etc/cloudera-scm-agent/config.ini and same issue occurs.

Quite certain I got the certs incorrectly configured. The reason why I'm thinking that is because I pointed agent.pem to various .pem files I have in the above-said folder. Same result.

I also have an HAproxy / Keepalived configuration pointing to a common VIP between the NN's called cm-c01.mws.mds.xyz . That works fine for the CM UI, however, I'm trying the agent connection to the primary server directly at first to get that working before trying via the VIP. So for the purpose of this exercise, I'm not using this VIP.

Here's my cert process:

1) Generate a cert.

keytool -genkeypair -alias $(hostname -f) -keyalg RSA -keysize 2048 -dname "cn=$(hostname -f),OU=MDS,O=MDS,L=Los Angeles,ST=California,C=US" -keypass $(hostname -f) -keystore $(hostname -f).keystore.jks -storepass $(hostname -f) -validity 3650 -ext EKU=serverAuth,clientAuth,codeSigning,emailProtection,timeStamping,OCSPSigning -ext san=dns:$(hostname -f)

2) Export cert from keystore.

keytool -export -alias $(hostname -f) -keystore $(hostname -f).keystore.jks -rfc -file $(hostname -f)-selfsigned.cer

3) Import the cert into the java truststores.

# Import root ca to the JDK Truststore
keytool -import -alias $(hostname -f) -file /opt/cloudera/security/jks/$(hostname -f)-selfsigned.cer -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit



# Import root ca to the JDK Truststore 
keytool -import -alias $(hostname -f) -file /opt/cloudera/security/jks/$(hostname -f)-selfsigned.cer -keystore /usr/java/jdk1.8.0_181-cloudera/jre/lib/security/jssecacerts -storepass changeit

4) Copy the files around and verify.

cp -ip /opt/cloudera/security/x509/$(hostname -f).pem /opt/cloudera/security/pki/$(hostname -f).pem;

cp -ip /opt/cloudera/security/x509/$(hostname -f).pem /opt/cloudera/security/pki/;
cp -ip /opt/cloudera/security/jks/$(hostname -f).keystore.jks /opt/cloudera/security/pki;

ln -s /opt/cloudera/security/pki/$(hostname -f).keystore.jks /opt/cloudera/security/pki/server.jks
ln -s /opt/cloudera/security/pki/$(hostname -f).pem /opt/cloudera/security/pki/agent.pem

openssl x509 -in /opt/cloudera/security/pki/$(hostname -f).pem -noout -text

5) Repeat the above on the other hosts.

6) HTTPS / 7183 works fine from CM, just not the agent to server communication, now that I have it enabled.

The parts I'm not clear about stems from my intermediate knowledge of certs:

A) Do I copy the generated cert off the CM / NN server to the other servers in the cluster?

B) Should I be exporting the private key from the SSL keystore by I) converting to pcs12 II) exporting the cert and key from that? One of the dozens of options I tried included this step but that didn't work out either when I was trying the connectivity of the cloudera-scm-agent on the NN instead of any other participating host:

keytool -importkeystore \
    -srckeystore cm-r01nn01.mws.mds.xyz.keystore.jks \
    -destkeystore cm-r01nn01.mws.mds.xyz.keystore.p12 \
    -deststoretype PKCS12 \
    -srcalias cm-c01.mws.mds.xyz \
    -deststorepass cm-r01nn01.mws.mds.xyz \
    -destkeypass cm-r01nn01.mws.mds.xyz
	
	
# openssl pkcs12 -in cm-r01nn01.mws.mds.xyz.keystore.p12  -nokeys -out cm-r01nn01.mws.mds.xyz.cert.pem
Enter Import Password:
MAC verified OK
#

# openssl pkcs12 -in cm-r01nn01.mws.mds.xyz.keystore.p12  -nodes -nocerts -out cm-r01nn01.mws.mds.xyz.key.pem
Enter Import Password:
MAC verified OK
#

Thinking I might have the commands right, just not the sequence or I'm missing a step or two. Further notes and info:

[ UTILITY NODE ]

[root@cm-r01en01 pki]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts|grep cm
Enter keystore password:  changeit
acraizfnmt-rcm, Mar 26, 2019, trustedCertEntry,
cm-r01nn02.mws.mds.xyz, Apr 14, 2019, trustedCertEntry,
cm-r01en01.mws.mds.xyz, Apr 1, 2019, trustedCertEntry,
cm-r01nn01.mws.mds.xyz, Apr 14, 2019, trustedCertEntry,
[root@cm-r01en01 pki]#

[ Name Node / CM Server]

[root@cm-r01nn01 pki]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts|grep cm
Enter keystore password:  changeit
acraizfnmt-rcm, Mar 26, 2019, trustedCertEntry,
cm-r01nn02.mws.mds.xyz, Apr 14, 2019, trustedCertEntry,
cm-c01.mws.mds.xyz, Jul 6, 2019, trustedCertEntry,
cm-r01nn01.mws.mds.xyz, Mar 31, 2019, trustedCertEntry,
[root@cm-r01nn01 pki]#

The only other message that I got when moving certs around under the agent.pem symlink, was:

SSLError: sslv3 alert bad certificate

I'm not clear if that error is a step forward or backwards. Also, discovered by accident that I may not need to restart the cloudera-scm-agent under every scenario. Just changing where the symlink agent.pem points too appear to generate different messages, even when the agent is running.

Thx,
TK

Re: SSLError: certificate verify failed

bgooley — Tue, 09 Jul 2019 16:49:48 GMT

@TCloud,

If you are using self-signed certificates, then it sounds as if you may have things right.

For the heartbeat issue, though, we can use openssl to see if we have things configured correctly and also see what public certificate is returned by Cloudera Manager for the heartbeat

Run:

# openssl s_client -connect cm-r01nn01.mws.mds.xyz:7182 -showcerts -CAfile /opt/cloudera/security/pki/cm-r01en01.mws.mds.xyz.pem

I believe you will still get an error now that you have explained some of the backstory.

When the agent is making a connection to CM to perform the heartbeat, it needs to do a TLS handshake per your configuration. Part of this handshake is the client (the agent) deciding if it trusts the certificate returned from CM.

In your config.ini, you are connecting to the CM host: cm-r01nn01.mws.mds.xyz

However, your pem file name indicates that you are pointing verify_cert_file to the agent's host's certificate, not the certificate of the Cloudera Manager host.

In essence, the agent needs to trust the certificate returned by CM in the handshake.

So, the verify_cert_file file will need to contain the certificate for cm-r01nn01.mws.mds.xyz

If that doesn't make sense, provide the output from the openssl command above and then also the output from:

# openssl x509 -in /opt/cloudera/security/pki/cm-r01en01.mws.mds.xyz.pem -text -noout

From that we can provide further advice.

Re: SSLError: certificate verify failed

bgooley — Tue, 09 Jul 2019 16:51:18 GMT

NOTE:

If you are trying to do agent authentication to CM, then you will need to configure the key/cert for the agent to present to CM. This is a separate, additional configuration than we have been discussing if you have configured Agent Authentication to CM in Cloudera Manager.

Re: SSLError: certificate verify failed

TCloud — Tue, 09 Jul 2019 17:00:51 GMT

Just briefly:

[root@cm-r01en01 pki]# openssl s_client -connect cm-r01nn01.mws.mds.xyz:7182 -showcerts -CAfile /opt/cloudera/security/pki/cm-r01en01.mws.mds.xyz.pem
CONNECTED(00000003)
depth=0 C = US, ST = California, L = Los Angeles, O = MDS, OU = MDS, CN = cm-r01nn01.mws.mds.xyz
verify return:1
140043391174544:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:s3_pkt.c:1493:SSL alert number 42
140043391174544:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:
---
Certificate chain
 0 s:/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-r01nn01.mws.mds.xyz
   i:/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-r01nn01.mws.mds.xyz
-----BEGIN CERTIFICATE-----
MIID9TCCAt2gAwIBAgIEVUJkFTANBgkqhkiG9w0BAQsFADB1MQswCQYDVQQGEwJV
UzETMBEGA1UECBMKQ2FsaWZvcm5pYTEUMBIGA1UEBxMLTG9zIEFuZ2VsZXMxDDAK
BgNVBAoTA01EUzEMMAoGA1UECxMDTURTMR8wHQYDVQQDExZjbS1yMDFubjAxLm13
cy5tZHMueHl6MB4XDTE5MDQwMTAyMzc1MFoXDTI5MDMyOTAyMzc1MFowdTELMAkG
A1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWExFDASBgNVBAcTC0xvcyBBbmdl
bGVzMQwwCgYDVQQKEwNNRFMxDDAKBgNVBAsTA01EUzEfMB0GA1UEAxMWY20tcjAx
bm4wMS5td3MubWRzLnh5ejCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AIVFcUGgk3fKC1zuKuSTmc6HQLIxvI9fZXjWMOwYagoJjAgKEFUKtjUTgY15IqT9
VXeBuOICTzDvXQbHEh87KAVLoyQa+DkBR9yoyj5hhRUz+fG7erVgvf1AyB5V289Q
ZZir70nzrnRfGdGh4gFCPH2CKvTBndvLYIbDlSh8P5gjLyck7KMW7GrbtZfxcTGD
qBJYB8okK6/Fs6fYD5UggfYIVhEgAorciHCYlnsJagSda/8Mn5a9WtYLjMJY8hEW
RjZxLLI6aunrd/J24ulUEURaWiqjtfna7Q++mDL4GLtf/sUe4xYjt5zy7qMWPh6Q
aGNVcR+bMVgs+e2qATPbiLkCAwEAAaOBjDCBiTBFBgNVHSUEPjA8BggrBgEFBQcD
AQYIKwYBBQUHAwIGCCsGAQUFBwMDBggrBgEFBQcDBAYIKwYBBQUHAwgGCCsGAQUF
BwMJMCEGA1UdEQQaMBiCFmNtLXIwMW5uMDEubXdzLm1kcy54eXowHQYDVR0OBBYE
FCE0pxrJ9cnfzdjidWWTm5YQFoZ/MA0GCSqGSIb3DQEBCwUAA4IBAQBe9m5Tg76b
Uux3yZudIAtISKwSGP2hz4kYpGEM5ykjiY4UAtScFHLcPEJxFN6K6KtZDy02nStI
aLE9dNB3SayLspC6nC1gAB05D8viFKniOVSG8TQgTytOHG/A7UWz5yQF4uJXiETe
de82xiMt75O1jaePFrsMG2twSZxRTLpd52WGit4A5B8LM2ADTR/wsFvQMhvZA+y5
N+1/pw8bQK4SGGWZ4DWXP+Q5bC/6xP5gN1H9bmvvZILNVma7w+ko7Wr8qIfRB8RU
2j1EiHAp99CjA1on1XVH//3TAhRhvXojh+YnZAwe4JCkCI6vVEYSkbgWBK1txYEa
uZw5Rx5pl+gE
-----END CERTIFICATE-----
---
Server certificate
subject=/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-r01nn01.mws.mds.xyz
issuer=/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-r01nn01.mws.mds.xyz
---
Acceptable client certificate CA names
/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-r01nn01.mws.mds.xyz
/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-c01.mws.mds.xyz
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1733 bytes and written 138 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 5D24C76A5F77382AA772B71FCEC240F1DB45EB8FF77E4B930D58C3CBBABE5682
    Session-ID-ctx:
    Master-Key: 20625D24BF6F4044D4E85E71FC57D350607611BD6F6EF42AA643208DBEB8F9CCFF8FB90B43512A0D548E5E0A385C2B3E
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1562691434
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
[root@cm-r01en01 pki]#

[root@cm-r01en01 pki]# openssl x509 -in /opt/cloudera/security/pki/cm-r01en01.mws.mds.xyz.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 155294363 (0x9419a9b)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, ST=California, L=Los Angeles, O=MDS, OU=MDS, CN=cm-r01en01.mws.mds.xyz
        Validity
            Not Before: Apr  2 03:06:32 2019 GMT
            Not After : Mar 30 03:06:32 2029 GMT
        Subject: C=US, ST=California, L=Los Angeles, O=MDS, OU=MDS, CN=cm-r01en01.mws.mds.xyz
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:a8:d6:e1:0d:25:38:62:1c:fc:26:2c:2d:d4:57:
                    f5:d9:54:f1:ee:8b:5d:45:d9:74:86:3d:08:b3:67:
                    4b:63:ef:7e:ae:f0:ae:84:92:1e:45:83:d8:54:f5:
                    0d:cf:ba:22:80:ed:96:30:60:37:d5:82:11:ed:8e:
                    9b:42:4b:81:1d:c1:5b:45:62:00:0d:20:d5:8b:51:
                    18:df:3f:2a:5a:33:dc:c5:85:63:18:ed:b8:0c:58:
                    a7:26:22:61:b3:16:51:c8:97:42:eb:10:0c:67:00:
                    d1:3c:24:eb:2d:08:7c:fb:91:57:d1:1c:8e:3b:81:
                    a9:e6:f4:6c:ff:1a:f6:7d:9a:07:a4:f1:47:50:ac:
                    44:16:d8:27:17:b2:02:2b:eb:4a:ef:1b:34:69:c2:
                    15:a6:92:a4:7b:0a:f0:c0:43:95:91:c0:f5:40:3e:
                    c6:6f:b2:db:e0:5f:9f:ed:10:36:26:db:d3:e1:d5:
                    42:48:19:73:8a:72:7e:2f:f8:92:89:de:1b:42:64:
                    c9:fa:80:a5:38:ad:c9:4f:e8:96:74:38:d5:58:2d:
                    14:31:36:51:22:14:fc:31:a2:74:3e:91:a4:b3:d5:
                    51:de:70:80:42:0f:0a:58:8b:11:e3:89:e3:a9:78:
                    4f:2e:9d:ae:ab:17:2f:fd:ff:9b:ee:a5:d9:d1:6f:
                    b6:71
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication, Code Signing, E-mail Protection, Time Stamping, OCSP Signing
            X509v3 Subject Alternative Name:
                DNS:cm-r01en01.mws.mds.xyz
            X509v3 Subject Key Identifier:
                41:99:DC:C2:1A:EC:17:F5:89:8B:1A:57:38:FE:FF:8B:28:28:41:F6
    Signature Algorithm: sha256WithRSAEncryption
         57:10:c8:55:05:f9:4b:a1:11:3b:90:19:7a:7b:b5:14:e7:f2:
         7c:d4:c9:c4:cc:49:27:4d:e0:28:d6:0d:f6:08:00:cf:45:c4:
         02:19:bc:2a:9c:bf:df:9c:44:db:00:25:51:af:0b:2d:ba:e9:
         68:eb:dd:c9:de:4a:0f:c9:62:7d:c2:b9:24:e9:1a:ce:bb:3d:
         53:11:9d:d7:6c:25:4c:10:b5:36:79:01:81:8f:7b:29:d4:bc:
         d0:60:7a:81:e9:ff:85:67:f2:62:b8:5c:29:fe:6c:68:72:e7:
         c7:8d:4e:eb:91:17:b1:c7:35:df:cd:cf:33:70:a5:55:4b:fc:
         0d:f1:45:d4:0e:b7:7a:03:53:8a:ae:50:3b:43:9b:04:7e:f0:
         ea:2a:da:7d:03:b7:5c:ce:63:44:c4:4c:17:41:02:cd:87:3e:
         19:a4:9b:4c:18:54:b3:3e:39:53:a2:33:fb:94:ad:e4:1c:86:
         4e:48:aa:85:86:ae:b6:5f:8c:73:81:26:47:a8:19:c8:9e:19:
         6d:0a:8d:70:68:f8:c2:26:c8:66:5b:80:c7:57:12:e7:f8:cb:
         78:3c:75:f1:d5:40:2b:64:87:23:9b:82:5a:70:5b:ed:a3:e1:
         78:43:5b:c7:59:72:aa:8b:a6:bc:c8:72:80:3d:22:85:94:3d:
         03:0d:c8:26
[root@cm-r01en01 pki]#

Will reread your post again in a few minutes.

Re: SSLError: certificate verify failed

TCloud — Tue, 09 Jul 2019 19:29:36 GMT

At one point I was receiving this error as well (which made sense to me):

http://gagravarr.org/writing/openssl-certs/errors.shtml

However, it did not help out with immediate issue above.

Re: SSLError: certificate verify failed

TCloud — Wed, 10 Jul 2019 00:42:36 GMT

Pointed agent.pem as follows:

[root@cm-r01en01 pki]# ls -altri
total 36
69943257 -rw-r--r--. 1 cloudera-scm cloudera-scm 2385 Apr  1 23:06 cm-r01en01.mws.mds.xyz.keystore.jks-backup
69943152 -rw-r--r--. 1 cloudera-scm cloudera-scm 1453 Apr  1 23:07 cm-r01en01.mws.mds.xyz.pem
 3870062 drwxr-xr-x. 5 root         root           37 Apr  1 23:09 ..
69943169 lrwxrwxrwx. 1 root         root           62 Apr  1 23:11 server.jks -> /opt/cloudera/security/pki/cm-r01en01.mws.mds.xyz.keystore.jks
69943259 -rw-r--r--. 1 cloudera-scm cloudera-scm 1453 Jul  6 20:01 cm-r01nn01.mws.mds.xyz.pem
69943154 lrwxrwxrwx. 1 root         root           53 Jul  6 20:02 rootca.pem -> /opt/cloudera/security/pki/cm-r01nn01.mws.mds.xyz.pem
69943167 -rw-r--r--. 1 cloudera-scm cloudera-scm 3449 Jul  8 03:32 cm-r01en01.mws.mds.xyz.keystore.jks
71202464 -rw-r--r--. 1 cloudera-scm cloudera-scm 1415 Jul  8 03:52 cm-r01nn01.mws.mds.xyz.cert.pem
71202517 -rw-r--r--. 1 cloudera-scm cloudera-scm 1859 Jul  8 03:53 cm-r01nn01.mws.mds.xyz.key.pem
71305734 -rw-r--r--. 1 cloudera-scm cloudera-scm 3119 Jul  8 03:56 cm-r01nn01.mws.mds.xyz.cert.key.pem
67257529 -rw-r--r--. 1 cloudera-scm cloudera-scm   23 Jul  8 03:57 agent.pw
67689060 lrwxrwxrwx. 1 root         root           53 Jul  9 20:21 agent.pem -> /opt/cloudera/security/pki/cm-r01nn01.mws.mds.xyz.pem
69943151 drwxr-xr-x. 2 root         root         4096 Jul  9 20:21 .
[root@cm-r01en01 pki]# pwd
/opt/cloudera/security/pki
[root@cm-r01en01 pki]#

Now the error is:

[09/Jul/2019 20:22:46 +0000] 19927 MainThread agent        ERROR    Heartbeating to cm-r01nn01.mws.mds.xyz:7182 failed.
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeat
    self.cfg.max_cert_depth)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__
    self.conn.connect()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect
    sock.connect((self.host, self.port))
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connect
    ret = self.connect_ssl()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_ssl
    return m2.ssl_connect(self.ssl, self._timeout)
SSLError: sslv3 alert bad certificate
^C
[root@cm-r01en01 pki]#

Now I tried to extract the key/cert from the NN generated certs like this (convert to pkcs12 then extract the key/cert):

[root@cm-r01nn01 pki]# ls -altri
total 32
135962830 -rw-r--r--. 1 root         root         1453 Mar 31 22:41 cm-r01nn01.mws.mds.xyz.pem
201424378 drwxr-xr-x. 5 root         root           37 Mar 31 23:42 ..
135962831 lrwxrwxrwx. 1 root         root           53 Mar 31 23:44 agent.pem -> /opt/cloudera/security/pki/cm-r01nn01.mws.mds.xyz.pem
135962833 lrwxrwxrwx. 1 root         root           62 Mar 31 23:45 server.jks -> /opt/cloudera/security/pki/cm-r01nn01.mws.mds.xyz.keystore.jks
135962832 -rw-r--r--. 1 cloudera-scm cloudera-scm 4723 Jul  6 00:00 cm-r01nn01.mws.mds.xyz.keystore.jks
135283830 -rw-r--r--. 1 root         root         1435 Jul  6 00:10 cm-c01.mws.mds.xyz.pem
135044185 -rw-r--r--. 1 root         root         2751 Jul  8 03:49 cm-r01nn01.mws.mds.xyz.keystore.p12
135044187 -rw-r--r--. 1 root         root         1691 Jul  8 03:50 cm-r01nn01.mws.mds.xyz.cert.pem
135962829 drwxr-xr-x. 2 root         root         4096 Jul  8 03:50 .
135044188 -rw-r--r--. 1 root         root         1859 Jul  8 03:50 cm-r01nn01.mws.mds.xyz.key.pem
[root@cm-r01nn01 pki]# pwd
/opt/cloudera/security/pki
[root@cm-r01nn01 pki]#

Then copied them above to the cm-r01en01.mws.mds.xyz client (as you probably noticed). Now I'm not clear on my next step. If I read you correctly, seems as if I need the client's key/cert instead?

Verifying using openssl yields the same error:

[root@cm-r01en01 pki]# openssl s_client -connect cm-r01nn01.mws.mds.xyz:7182 < cm-r01nn01.mws.mds.xyz.pem
CONNECTED(00000003)
depth=0 C = US, ST = California, L = Los Angeles, O = MDS, OU = MDS, CN = cm-r01nn01.mws.mds.xyz
verify return:1
139746469861264:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:s3_pkt.c:1493:SSL alert number 42
139746469861264:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:
---
Certificate chain
 0 s:/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-r01nn01.mws.mds.xyz
   i:/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-r01nn01.mws.mds.xyz
---
Server certificate
-----BEGIN CERTIFICATE-----
MIID9TCCAt2gAwIBAgIEVUJkFTANBgkqhkiG9w0BAQsFADB1MQswCQYDVQQGEwJV
UzETMBEGA1UECBMKQ2FsaWZvcm5pYTEUMBIGA1UEBxMLTG9zIEFuZ2VsZXMxDDAK
BgNVBAoTA01EUzEMMAoGA1UECxMDTURTMR8wHQYDVQQDExZjbS1yMDFubjAxLm13
cy5tZHMueHl6MB4XDTE5MDQwMTAyMzc1MFoXDTI5MDMyOTAyMzc1MFowdTELMAkG
A1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWExFDASBgNVBAcTC0xvcyBBbmdl
bGVzMQwwCgYDVQQKEwNNRFMxDDAKBgNVBAsTA01EUzEfMB0GA1UEAxMWY20tcjAx
bm4wMS5td3MubWRzLnh5ejCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AIVFcUGgk3fKC1zuKuSTmc6HQLIxvI9fZXjWMOwYagoJjAgKEFUKtjUTgY15IqT9
VXeBuOICTzDvXQbHEh87KAVLoyQa+DkBR9yoyj5hhRUz+fG7erVgvf1AyB5V289Q
ZZir70nzrnRfGdGh4gFCPH2CKvTBndvLYIbDlSh8P5gjLyck7KMW7GrbtZfxcTGD
qBJYB8okK6/Fs6fYD5UggfYIVhEgAorciHCYlnsJagSda/8Mn5a9WtYLjMJY8hEW
RjZxLLI6aunrd/J24ulUEURaWiqjtfna7Q++mDL4GLtf/sUe4xYjt5zy7qMWPh6Q
aGNVcR+bMVgs+e2qATPbiLkCAwEAAaOBjDCBiTBFBgNVHSUEPjA8BggrBgEFBQcD
AQYIKwYBBQUHAwIGCCsGAQUFBwMDBggrBgEFBQcDBAYIKwYBBQUHAwgGCCsGAQUF
BwMJMCEGA1UdEQQaMBiCFmNtLXIwMW5uMDEubXdzLm1kcy54eXowHQYDVR0OBBYE
FCE0pxrJ9cnfzdjidWWTm5YQFoZ/MA0GCSqGSIb3DQEBCwUAA4IBAQBe9m5Tg76b
Uux3yZudIAtISKwSGP2hz4kYpGEM5ykjiY4UAtScFHLcPEJxFN6K6KtZDy02nStI
aLE9dNB3SayLspC6nC1gAB05D8viFKniOVSG8TQgTytOHG/A7UWz5yQF4uJXiETe
de82xiMt75O1jaePFrsMG2twSZxRTLpd52WGit4A5B8LM2ADTR/wsFvQMhvZA+y5
N+1/pw8bQK4SGGWZ4DWXP+Q5bC/6xP5gN1H9bmvvZILNVma7w+ko7Wr8qIfRB8RU
2j1EiHAp99CjA1on1XVH//3TAhRhvXojh+YnZAwe4JCkCI6vVEYSkbgWBK1txYEa
uZw5Rx5pl+gE
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-r01nn01.mws.mds.xyz
issuer=/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-r01nn01.mws.mds.xyz
---
Acceptable client certificate CA names
/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-r01nn01.mws.mds.xyz
/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-c01.mws.mds.xyz
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1733 bytes and written 138 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 5D25323127BD39A3A3E9C85D12EA95BB46D7EF0D1A2D5EBCD68D502DC140431F
    Session-ID-ctx:
    Master-Key: 96673D16341D315F828D023D34D65937D8BCBA58437F9926B74450998EF22C3CBC038BD099F815CFC6053F4630490362
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1562718769
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
[root@cm-r01en01 pki]#

Admin -> Settings:

Use TLS Encryption for Admin Console (Enabled)

Use TLS Encryption for Agents (Enabled)
Use TLS Authentication of Agents to Server (Enabled)

Verify Agent Hostname Against Certificate (Enabled)

Cloudera Manager TLS/SSL Server JKS Keystore File Location = /opt/cloudera/security/pki/server.jks
Cloudera Manager TLS/SSL Server JKS Keystore File Password = <PASS = cm-r01nn01.mws.mds.xyz>

Supported SSL/TLS versions = SSLv2Hello, and TLSv1.2

CMS -> Configuration

TLS/SSL Client Truststore File Location = /etc/pki/ca-trust/extracted/java/jssecacerts

Cloudera Manager Server TLS/SSL Client Trust Store Password = <PASS = changeit >

Options 2-3 above are the two that I don't fully grasp yet.

Cheers,
TK

Re: SSLError: certificate verify failed

bgooley — Wed, 10 Jul 2019 01:09:14 GMT

@TCloud,

First, some background on what the settings do:

(1)

Use TLS Encryption for Agents - This means that Cloudera Manager's heartbeat port (7182) will require TLS. All agents need in order to use tls is the following:

[Security]
use_tls=1

(2)

If you then want the agents to verify the Cloudera Manager certificate, you can then add the verify_cert_file configuration:

[Security]
use_tls=1

verify_cert_file=/path/to/truststore

The "verify_cert_file" value is a path to a file that contains one or more certfificates trusted by the agent. If you are using self-signed certificates, you can simply use the PEM format of the CM server's certificate. In your case, the file needs to contain the following:

(3)

Use TLS Authentication of Agents to Server - This means that Cloudera Manager requires that any agent heartbeating will supply its public certificate as means of authenticating to Cloudera Manager.

Agent requirements:

Along with the previous configuration, you need the following:

client_key_file=/path/to/private/key/file
client_keypw_file=/path/to/file/containing/password/for/private/key/file
client_cert_file=/path/to/server/certificate/paired/with/agent/private/key

For example:

[Security]
use_tls=1
max_cert_depth=9
verify_cert_file=/opt/cloudera/security/cacerts/ClouderaSEC_combined.pem
client_key_file=/opt/cloudera/security/x509/host.key
client_keypw_file=/opt/cloudera/security/agent_key_pw
client_cert_file=/opt/cloudera/security/x509/host.pem

Cloudera Manager requirement:

CM truststore must contain the signer of each agent's certificate

CM must be configured with a trust store in "Cloudera Manager TLS/SSL Client Trust Store File"

It appears this file may contain all the hosts' agent certificates:

/etc/pki/ca-trust/extracted/java/jssecacerts

NEXT STEPS:

It appears that your agents are not configured to perform agent authentication so Cloudera Manager is returning the Bad Certificate TLS Alert and failing the TLS Handshake. This is a good sign as it means the agent seems to now be able to trust the signer of your CM certificate.

To correct the situation, you need to make sure that you configure client_key_file, client_keypw_file, and client_cert_file

Make sure to restart the agent after adding them.

Re: SSLError: certificate verify failed

TCloud — Fri, 12 Jul 2019 02:47:20 GMT

Got steps 1 and 2 working. Tyvm.

For step 3, I'm working on it now in combination with your post above and noticing this page might not be correct:

https://www.cloudera.com/documentation/enterprise/latest/topics/how_to_configure_cm_tls.html#concept_xtp_q3w_wn

"Cloudera Manager TLS/SSL Certificate Trust Store File" doesn't appear under Administration -> Settings.

Thx,
TK

Re: SSLError: certificate verify failed

TCloud — Fri, 12 Jul 2019 03:52:36 GMT

The error messages I get from the agent when I attempt (3):

==> /var/log/cloudera-scm-agent/status-stderr.log <==
[11/Jul/2019:23:37:40] ENGINE Error in HTTPServer.tick
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cheroot/server.py", line 1339, in start
self.tick()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cheroot/server.py", line 1408, in tick
s, ssl_env = self.ssl_adapter.wrap(s)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/status_server.py", line 1048, in wrap
ssl.accept_ssl()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 258, in accept_ssl
return m2.ssl_accept(self.ssl, self._timeout)
SSLError: unexpected eof

[11/Jul/2019 23:37:43 +0000] 8193 MainThread agent        ERROR    Heartbeating to cm-r01nn01.mws.mds.xyz:7182 failed.
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeat
    self.cfg.max_cert_depth)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__
    self.conn.connect()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect
    sock.connect((self.host, self.port))
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connect
    ret = self.connect_ssl()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_ssl
    return m2.ssl_connect(self.ssl, self._timeout)
SSLError: sslv3 alert certificate unknown

The pressing question I have above all others is how do I get the agent to print out enough logging to tell me WHICH certificate it attempted to load so I know the context under which the above is thrown? Right now, given the above error, I can't really take action without knowing the exact file the exceptions are referring too, other then deduce based on the change I've done.

My setup as it is now:

[ cm-r01en01 ] ( Utility Server )

[root@cm-r01en01 pki]# ls -altri
total 32
 3870062 drwxr-xr-x. 5 root         root           37 Apr  1 23:09 ..
69943169 lrwxrwxrwx. 1 root         root           62 Apr  1 23:11 server.jks -> /opt/cloudera/security/pki/cm-r01en01.mws.mds.xyz.keystore.jks
69943167 -rw-r--r--. 1 cloudera-scm cloudera-scm 3449 Jul  8 03:32 cm-r01en01.mws.mds.xyz.keystore.jks
67257528 -rw-r--r--. 1 cloudera-scm cloudera-scm 2775 Jul  9 23:51 cm-r01en01.mws.mds.xyz.keystore.p12
67586231 -rw-r--r--. 1 cloudera-scm cloudera-scm 1720 Jul  9 23:52 cm-r01en01.mws.mds.xyz.cert.pem
71202518 -rw-r--r--. 1 cloudera-scm cloudera-scm 1863 Jul  9 23:53 cm-r01en01.mws.mds.xyz.key.pem
71305735 -r--r-----. 1 cloudera-scm cloudera-scm   23 Jul  9 23:53 client-agent.pw
67257529 lrwxrwxrwx. 1 root         root           30 Jul  9 23:55 client-key.pem -> cm-r01en01.mws.mds.xyz.key.pem
71202519 lrwxrwxrwx. 1 root         root           31 Jul  9 23:55 client-cert.pem -> cm-r01en01.mws.mds.xyz.cert.pem
71305736 -rw-r--r--. 1 cloudera-scm cloudera-scm 1432 Jul 10 20:27 cm-r01en01.mws.mds.xyz.pem
71305755 -rw-r--r--. 1 cloudera-scm cloudera-scm 1453 Jul 10 22:14 cm-r01nn01.mws.mds.xyz.pem
69943176 lrwxrwxrwx. 1 root         root           53 Jul 10 22:14 agent.pem -> /opt/cloudera/security/pki/cm-r01nn01.mws.mds.xyz.pem
69943151 drwxr-xr-x. 2 root         root         4096 Jul 11 23:01 .
[root@cm-r01en01 pki]# hostname
cm-r01en01.mws.mds.xyz
[root@cm-r01en01 pki]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts|grep -Ei cm
Enter keystore password:  changeit
acraizfnmt-rcm, Mar 26, 2019, trustedCertEntry,
cm-r01nn02.mws.mds.xyz, Apr 14, 2019, trustedCertEntry,
cm-r01en01.mws.mds.xyz, Jul 11, 2019, trustedCertEntry,
cm-r01nn01.mws.mds.xyz, Apr 14, 2019, trustedCertEntry,
[root@cm-r01en01 pki]#

[root@cm-r01en01 pki]# cat /etc/cloudera-scm-agent/config.ini|grep -Eiv "#"|sed -e "/^$/d"
[General]
server_host=cm-r01nn01.mws.mds.xyz
server_port=7182
max_collection_wait_seconds=10.0
metrics_url_timeout_seconds=30.0
task_metrics_timeout_seconds=5.0
monitored_nodev_filesystem_types=nfs,nfs4,tmpfs
local_filesystem_whitelist=ext2,ext3,ext4,xfs
impala_profile_bundle_max_bytes=1073741824
stacks_log_bundle_max_bytes=1073741824
stacks_log_max_uncompressed_file_size_bytes=5242880
orphan_process_dir_staleness_threshold=5184000
orphan_process_dir_refresh_interval=3600
scm_debug=logging.DEBUG
dns_resolution_collection_interval_seconds=60
dns_resolution_collection_timeout_seconds=30
[Security]
use_tls=1
max_cert_depth=9
verify_cert_file=/opt/cloudera/security/pki/agent.pem
client_key_file=/opt/cloudera/security/pki/client-key.pem
client_keypw_file=/opt/cloudera/security/pki/client-agent.pw
client_cert_file=/opt/cloudera/security/pki/client-cert.pem
[Hadoop]
[Cloudera]
[JDBC]
[Cgroup_Paths]
[root@cm-r01en01 pki]#

[ cm-r01en02 ] ( Utility Server )

[root@cm-r01en02 pki]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts|grep -Ei cm
Enter keystore password:  changeit
acraizfnmt-rcm, Mar 26, 2019, trustedCertEntry,
cm-r01nn02.mws.mds.xyz, Jul 10, 2019, trustedCertEntry,
cm-r01en02.mws.mds.xyz, Jul 10, 2019, trustedCertEntry,
cm-r01nn01.mws.mds.xyz, Jul 10, 2019, trustedCertEntry,
You have new mail in /var/spool/mail/root
[root@cm-r01en02 pki]# ls -altri
total 28
135616270 drwxr-xr-x. 5 root         root           37 Jul 10 21:28 ..
335605256 -rw-r--r--. 1 cloudera-scm cloudera-scm 2386 Jul 10 21:29 cm-r01en02.mws.mds.xyz.keystore.jks
335605249 -rw-r--r--. 1 cloudera-scm cloudera-scm 1453 Jul 10 21:51 cm-r01en02.mws.mds.xyz.pem
335605265 lrwxrwxrwx. 1 root         root           62 Jul 10 21:56 server.jks -> /opt/cloudera/security/pki/cm-r01en02.mws.mds.xyz.keystore.jks
335605275 -rw-r--r--. 1 cloudera-scm cloudera-scm 1453 Jul 10 22:14 cm-r01nn01.mws.mds.xyz.pem
335605382 lrwxrwxrwx. 1 root         root           53 Jul 10 22:14 agent.pem -> /opt/cloudera/security/pki/cm-r01nn01.mws.mds.xyz.pem
335605420 -rw-r--r--. 1 cloudera-scm cloudera-scm 2775 Jul 11 23:09 cm-r01en02.mws.mds.xyz.keystore.p12
335605426 -rw-r--r--. 1 cloudera-scm cloudera-scm 1720 Jul 11 23:10 cm-r01en02.mws.mds.xyz.cert.pem
335605425 -rw-r--r--. 1 cloudera-scm cloudera-scm 1863 Jul 11 23:11 cm-r01en02.mws.mds.xyz.key.pem
335605429 lrwxrwxrwx. 1 root         root           30 Jul 11 23:12 client-key.pem -> cm-r01en02.mws.mds.xyz.key.pem
335605430 lrwxrwxrwx. 1 root         root           31 Jul 11 23:12 client-cert.pem -> cm-r01en02.mws.mds.xyz.cert.pem
335860926 drwxr-xr-x. 2 root         root         4096 Jul 11 23:12 .
[root@cm-r01en02 pki]# cat /etc/cloudera-scm-agent/config.ini|grep -Eiv "#"|sed -e "/^$/d"
[General]
server_host=cm-r01nn01.mws.mds.xyz
server_port=7182
max_collection_wait_seconds=10.0
metrics_url_timeout_seconds=30.0
task_metrics_timeout_seconds=5.0
monitored_nodev_filesystem_types=nfs,nfs4,tmpfs
local_filesystem_whitelist=ext2,ext3,ext4,xfs
impala_profile_bundle_max_bytes=1073741824
stacks_log_bundle_max_bytes=1073741824
stacks_log_max_uncompressed_file_size_bytes=5242880
orphan_process_dir_staleness_threshold=5184000
orphan_process_dir_refresh_interval=3600
scm_debug=logging.DEBUG
dns_resolution_collection_interval_seconds=60
dns_resolution_collection_timeout_seconds=30
[Security]
use_tls=1
max_cert_depth=9
verify_cert_file=/opt/cloudera/security/pki/agent.pem
client_key_file=/opt/cloudera/security/pki/client-key.pem
client_keypw_file=/opt/cloudera/security/pki/agent.pw
client_cert_file=/opt/cloudera/security/pki/client-cert.pem
[Hadoop]
[Cloudera]
[JDBC]
[Cgroup_Paths]
You have new mail in /var/spool/mail/root
[root@cm-r01en02 pki]#

[ cm-r01nn01 ] (Name Node)

[root@cm-r01nn01 pki]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts|grep -Ei cm                       Enter keystore password:  changeit
acraizfnmt-rcm, Mar 26, 2019, trustedCertEntry,
cm-r01nn02.mws.mds.xyz, Apr 14, 2019, trustedCertEntry,
cm-c01.mws.mds.xyz, Jul 6, 2019, trustedCertEntry,
cm-r01nn01.mws.mds.xyz, Mar 31, 2019, trustedCertEntry,
[root@cm-r01nn01 pki]#
[root@cm-r01nn01 pki]#
[root@cm-r01nn01 pki]# cd /opt/cloudera/security/pki
[root@cm-r01nn01 pki]# ls -altri
total 32
201424378 drwxr-xr-x. 5 root         root           37 Mar 31 23:42 ..
135962833 lrwxrwxrwx. 1 root         root           62 Mar 31 23:45 server.jks -> /opt/cloudera/security/pki/cm-r01nn01.mws.mds.xyz.keystore.jks
135962832 -rw-r--r--. 1 cloudera-scm cloudera-scm 4723 Jul  6 00:00 cm-r01nn01.mws.mds.xyz.keystore.jks
135283830 -rw-r--r--. 1 root         root         1435 Jul  6 00:10 cm-c01.mws.mds.xyz.pem
135044185 -rw-r--r--. 1 root         root         2751 Jul  8 03:49 cm-r01nn01.mws.mds.xyz.keystore.p12
135044187 -rw-r--r--. 1 root         root         1691 Jul  8 03:50 cm-r01nn01.mws.mds.xyz.cert.pem
135044188 -rw-r--r--. 1 root         root         1859 Jul  8 03:50 cm-r01nn01.mws.mds.xyz.key.pem
135962830 -rw-r--r--. 1 root         root         1453 Jul 10 22:14 cm-r01nn01.mws.mds.xyz.pem
135962831 lrwxrwxrwx. 1 root         root           53 Jul 10 22:14 agent.pem -> /opt/cloudera/security/pki/cm-r01nn01.mws.mds.xyz.pem
135962829 drwxr-xr-x. 2 root         root         4096 Jul 10 22:14 .
[root@cm-r01nn01 pki]# cat /etc/cloudera-scm-agent/config.ini|grep -Eiv "#"|sed -e "/^$/d"
[General]
server_host=cm-r01nn01.mws.mds.xyz
server_port=7182
max_collection_wait_seconds=10.0
metrics_url_timeout_seconds=30.0
task_metrics_timeout_seconds=5.0
monitored_nodev_filesystem_types=nfs,nfs4,tmpfs
local_filesystem_whitelist=ext2,ext3,ext4,xfs
impala_profile_bundle_max_bytes=1073741824
stacks_log_bundle_max_bytes=1073741824
stacks_log_max_uncompressed_file_size_bytes=5242880
orphan_process_dir_staleness_threshold=5184000
orphan_process_dir_refresh_interval=3600
scm_debug=logging.DEBUG
dns_resolution_collection_interval_seconds=60
dns_resolution_collection_timeout_seconds=30
[Security]
use_tls=1
max_cert_depth=9
verify_cert_file=/opt/cloudera/security/pki/agent.pem
[Hadoop]
[Cloudera]
[JDBC]
[Cgroup_Paths]
[root@cm-r01nn01 pki]#

NOTE: I haven't reconfigured the /etc/cloudera-scm-agent/config.ini yet on this (NN) node since the two other nodes (EN) aren't working anyway.

Thx,
TK

Re: SSLError: certificate verify failed

bgooley — Fri, 12 Jul 2019 18:18:10 GMT

Hi @TCloud ,

The documentation contains information about the trusstore in Step 4:

https://www.cloudera.com/documentation/enterprise/latest/topics/how_to_configure_cm_tls.html#concept_xtp_q3w_wn

It does not include *what* to put in the truststore file, though, so yes, that is something that should be improved.

If you are using a Certificate Authority to sign your certificates, you can simply add the root CA certificate to the truststore.

If you are using self-signed certificates, each agent's certificate needs to be imported so CM can validate the agent's certificate.

Re: SSLError: certificate verify failed

TCloud — Tue, 16 Jul 2019 04:59:29 GMT

Thanks very much again for taking the time and explain here. I've got this part working as well.

In the meantime, I'm looking to enable high availability on the cluster. Have a few questions in this regard.

1) HAproxy is given as an example. I've used Haproxy + Keepalived for the CMS (7183) and a custom DNS entry, cm-c01.mws.mds.xyz to point to the cluster VIP. Everything works, including the certs for the UI. UI correctly displays the SSL certs for cm-c01 rather then the constituent hosts providing the backend.

2) I tried the same process with the Agent Avro port 7182 (If I'm calling it that correctly). I've set up a VIP, configured HAproxy and the proper SSL certs for srv-c01.mws.mds.xyz. This doesn't work. I've imported the right certs into the jssecerts file as well.

HAproxy config:

frontend cmin
        bind    cm-c01:443 ssl crt /etc/haproxy/certs/cm-c01.mws.mds.xyz-haproxy.pem no-sslv3
        default_backend cmback

backend cmback
        mode http
        balance roundrobin

        server cm-r01nn01.mws.mds.xyz cm-r01nn01.mws.mds.xyz:7183        ssl check verify none port 7183 inter 12000 rise 3 fall 3
        server cm-r01nn02.mws.mds.xyz cm-r01nn02.mws.mds.xyz:7183        ssl check verify none port 7183 inter 12000 rise 3 fall 3

frontend srvin
        log                         127.0.0.1           local0          debug
        bind                        srv-c01:17182       ssl crt /etc/haproxy/certs/srv-c01.mws.mds.xyz-haproxy.pem no-sslv3
        default_backend             srvback


backend srvback
        log /dev/log local0 debug
        mode http
        balance roundrobin

        server      cm-r01nn01.mws.mds.xyz      cm-r01nn01.mws.mds.xyz:7182 ssl check verify none port 7182 inter 12000 rise 3 fall 3
        server      cm-r01nn02.mws.mds.xyz      cm-r01nn02.mws.mds.xyz:7182 ssl check verify none port 7182 inter 12000 rise 3 fall 3

Each PEM file has a private and public key.

How could I get this working with Self-Signed SSL certs? Do I need to revert the HAproxy config to a tcp pass-through? If so, how will I handle the case when it fails over and needs the cert file of the second host? Do I combine the public pem file certs?

Thx,
TK

Re: SSLError: certificate verify failed

bgooley — Tue, 16 Jul 2019 18:30:12 GMT

@TCloud,

Can you clarify the problem you are seeing regarding port 7182?

Re: SSLError: certificate verify failed

TCloud — Wed, 17 Jul 2019 03:41:03 GMT

How do I get SSL to work properly with CM and port 7182 through a VIP provided by a Load Balancer?

A visualization:

https://ibb.co/hY0GsVY

Re: SSLError: certificate verify failed

bgooley — Wed, 17 Jul 2019 17:37:21 GMT

@TCloud,

Have you tried it and it failed?

If so, what was the problem.

You configure the agent with a hostname and a port that it will use to send heartbeats to that host and port.

If you have TLS enabled, then the same rules apply:

If the client (agent) is doing validation, then it must be able to trust the signer of the CM certificate and it must be able to validate that the hostname it connected to is included in the certificate (in Subject Alt Name or CN subject).

If you are doing agent authentication to CM, then CM must trust the signer of the certificate presented by the agent.

I don't know if TLS termination at the balancer will work unless the balancer can authenticate. I'd recommend against termination with heartbeats.

Re: SSLError: certificate verify failed

TCloud — Wed, 17 Jul 2019 22:42:32 GMT

Tried. Certificates appeared fine ( were recognized ). The issue appears to be between the Load Balancer VIP and HAproxy right now since I get this:

[17/Jul/2019 03:56:22 +0000] 20834 MainThread agent        ERROR    Heartbeating to srv-c01.mws.mds.xyz:17182 failed.
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1396, in _send_heartbeat
    response = self.requestor.request('heartbeat', heartbeat_data)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 141, in request
    return self.issue_request(call_request, message_name, request_datum)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 254, in issue_request
    call_response = self.transceiver.transceive(call_request)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 483, in transceive
    result = self.read_framed_message()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 489, in read_framed_message
    framed_message = response_reader.read_framed_message()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 417, in read_framed_message
    raise ConnectionClosedException("Reader read 0 bytes.")
ConnectionClosedException: Reader read 0 bytes.

This is really telling me, and please keep me honest, that the traffic isn't being passed between the VIP and the backend servers defined in the HAproxy config (see below).

Currently trying to solve the above by incorporating what this page mentions, however, I need a newer HAproxy version since mine doesn't support SNI.

My latest HAproxy config is as follows to try and solve the above issue by setting up HAproxy in TLS bridging mode:

frontend srvin
        log                         127.0.0.1           local0          debug
        bind                        srv-c01:17182       ssl crt /etc/haproxy/certs/srv-c01.mws.mds.xyz-haproxy.pem no-sslv3
        default_backend             srvback


backend srvback
        log /dev/log local0 debug
        mode http
        balance roundrobin
        cookie srv-c01 insert indirect nocache

        server      cm-r01nn01.mws.mds.xyz      cm-r01nn01.mws.mds.xyz:7182 ssl check verify none port 7182 inter 12000 rise 3 fall 3 cookie cm-r01nn01.mws.mds.xyz sni req.hdr(host)
        server      cm-r01nn02.mws.mds.xyz      cm-r01nn02.mws.mds.xyz:7182 ssl check verify none port 7182 inter 12000 rise 3 fall 3 cookie cm-r01nn02.mws.mds.xyz sni req.hdr(host)

Current error is as follows but that's an HAproxy problem now that I'm following up on separately:

[ALERT] 197/040530 (7560) : parsing [/etc/haproxy/haproxy.cfg:69] : 'server cm-r01nn02.mws.mds.xyz' unknown keyword 'sni'.

Question I had above is, is there any other page other than the following that demonstrates the use of a VIP w/ HAproxy and TLS config when load balancing Cloudera services?

Thx,
TK

Re: SSLError: certificate verify failed

TCloud — Thu, 18 Jul 2019 01:47:18 GMT

Breakdown of what I'm getting with different configs:

frontend srvin
        log                         127.0.0.1           local0          debug
        bind                        srv-c01:17182       ssl crt /etc/haproxy/certs/srv-c01.mws.mds.xyz-haproxy.pem no-sslv3
        option tcplog
        default_backend             srvback


backend srvback
        log /dev/log local0 debug
        mode tcp
        option tcplog
        balance roundrobin

        server      cm-r01nn01.mws.mds.xyz      cm-r01nn01.mws.mds.xyz:7182 check
        server      cm-r01nn02.mws.mds.xyz      cm-r01nn02.mws.mds.xyz:7182 check

Results in:

[17/Jul/2019 21:12:12 +0000] 25588 MainThread agent        ERROR    Heartbeating to srv-c01.mws.mds.xyz:17182 failed.
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1396, in _send_heartbeat
    response = self.requestor.request('heartbeat', heartbeat_data)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 141, in request
    return self.issue_request(call_request, message_name, request_datum)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 254, in issue_request
    call_response = self.transceiver.transceive(call_request)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 483, in transceive
    result = self.read_framed_message()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 489, in read_framed_message
    framed_message = response_reader.read_framed_message()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 417, in read_framed_message
    raise ConnectionClosedException("Reader read 0 bytes.")
ConnectionClosedException: Reader read 0 bytes.

frontend srvin
        log                         127.0.0.1           local0          debug
        bind                        srv-c01:17182
        option tcplog
        default_backend             srvback


backend srvback
        log /dev/log local0 debug
        mode tcp
        option tcplog
        balance roundrobin

        server      cm-r01nn01.mws.mds.xyz      cm-r01nn01.mws.mds.xyz:7182 check
        server      cm-r01nn02.mws.mds.xyz      cm-r01nn02.mws.mds.xyz:7182 check

Results in:

[17/Jul/2019 21:15:23 +0000] 25588 MainThread agent        ERROR    Heartbeating to srv-c01.mws.mds.xyz:17182 failed.
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeat
    self.cfg.max_cert_depth)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__
    self.conn.connect()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect
    sock.connect((self.host, self.port))
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 313, in connect
    if not check(self.get_peer_cert(), self.addr[0]):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Checker.py", line 125, in __call__
    fieldName='subjectAltName')
WrongHost: Peer certificate subjectAltName does not match host, expected srv-c01.mws.mds.xyz, got DNS:cm-r01nn01.mws.mds.xyz

frontend srvin
        log                         127.0.0.1           local0          debug
        bind                        srv-c01:17182       ssl crt /etc/haproxy/certs/srv-c01.mws.mds.xyz-haproxy.pem no-sslv3
        default_backend             srvback


backend srvback
        log /dev/log local0 debug
        mode http
        balance roundrobin
        cookie srv-c01 insert indirect nocache

        server      cm-r01nn01.mws.mds.xyz      cm-r01nn01.mws.mds.xyz:7182 ssl check verify none port 7182 inter 12000 rise 3 fall 3 cookie cm-r01nn01.mws.mds.xyz sni req.hdr(host)
        server      cm-r01nn02.mws.mds.xyz      cm-r01nn02.mws.mds.xyz:7182 ssl check verify none port 7182 inter 12000 rise 3 fall 3 cookie cm-r01nn02.mws.mds.xyz sni req.hdr(host)

Results in:

parsing [/etc/haproxy/haproxy.cfg:69] : 'server cm-r01nn02.mws.mds.xyz' unknown keyword 'sni'.

Config uses a pem file with certs from the two nodes + the VIP ( concatenated from srv-c01, cm-r01nn01 and cm-r01nn02 ). The certs appear to work ok. The traffic isn't passing through, however.

Config looks like this:

[root@cm-r01wn08 pki]# cat /etc/cloudera-scm-agent/config.ini|grep -v "#" | sed -e "/^$/d"
[General]
server_host=srv-c01.mws.mds.xyz
server_port=17182
max_collection_wait_seconds=10.0
metrics_url_timeout_seconds=30.0
task_metrics_timeout_seconds=5.0
monitored_nodev_filesystem_types=nfs,nfs4,tmpfs
local_filesystem_whitelist=ext2,ext3,ext4,xfs
impala_profile_bundle_max_bytes=1073741824
stacks_log_bundle_max_bytes=1073741824
stacks_log_max_uncompressed_file_size_bytes=5242880
orphan_process_dir_staleness_threshold=5184000
orphan_process_dir_refresh_interval=3600
scm_debug=DEBUG
dns_resolution_collection_interval_seconds=60
dns_resolution_collection_timeout_seconds=30
[Security]
use_tls=1
max_cert_depth=9
verify_cert_file=/opt/cloudera/security/pki/cluster-vip.pem
client_key_file=/opt/cloudera/security/pki/client-key.pem
client_keypw_file=/opt/cloudera/security/pki/agent.pw
client_cert_file=/opt/cloudera/security/pki/client-cert.pem
[Hadoop]
[Cloudera]
[JDBC]
[Cgroup_Paths]
[root@cm-r01wn08 pki]#

I would like to keep it pointed to the VIP because this guarantees the config will be identical on all hosts and ready for any kind of failover.

Thx,
TK

Re: SSLError: certificate verify failed

bgooley — Thu, 18 Jul 2019 05:09:11 GMT

@TCloud,

The configuration we want is the one that got us the following:

WrongHost: Peer certificate subjectAltName does not match host, expected srv-c01.mws.mds.xyz, got DNS:cm-r01nn01.mws.mds.xyz

This error means that the Cloudera Manager certificate only contains a SAN or CN subject value of cm-r01nn01.mws.mds.xyz. Since the agent is configured to connect to srv-c01.mws.mds.xyz, it attempts to validate that the certificate is valid for srv-c01.mws.mds.xyz.

This situation is addressed here:

https://www.cloudera.com/documentation/enterprise/latest/topics/admin_cm_ha_tls.html#cloudera-manager-server-cert-requirements-for-HA

In order to make sure that clients can connect to CM by using both srv-c01.mws.mds.xyz and cm-r01nn01.mws.mds.xyz, we need to create a self-signed certificate that contains both in Subject Alternative Name.

For a self-signed certificate, you could use:

keytool -keystore testkeystore.jks -storepass password -keypass password -alias cm-r01nn01.mws.mds.xyz -genkeypair -keysize 2048 -keyalg RSA -dname "CN=cm-r01nn01.mws.mds.xyz" -ext san=dns:cm-r01nn01.mws.mds.xyz,dns:srv-c01.mws.mds.xyz

If you do recreate the CM certificate like that, you will need to also replace the previous certifiate with this one in any trust store you created since a new key pair was created.

Although it might require a bit more doing, the above should address the error you get when using TLS pass-through in HAProxy. Next, we need to make sure that HAProxy routes requests to your primary CM host every time and only routes to the other host in the event of the primary host's failure. I believe this can be achieved by removing "balance roundrobin" but I'm not sure. I feel like it may make sense to use "backup" directives in the server configuration for nn02 but I'm not sure... seems our example doesn't feel it is necessary.

Re: SSLError: certificate verify failed

TCloud — Mon, 22 Jul 2019 05:01:35 GMT

Received the subject error when I replaced the certs with the SAN one that contained 3 hosts:

keytool -genkeypair -alias cm-c01.mws.mds.xyz -keyalg RSA -keysize 2048 -dname "cn=cm-c01.mws.mds.xyz,OU=MDS,O=MDS,L=Los Angeles,ST=California,C=US" -keypass cm-c01.mws.mds.xyz -keystore cm-c01.mws.mds.xyz.keystore.jks -storepass cm-c01.mws.mds.xyz -validity 3650 -ext EKU=serverAuth,clientAuth,codeSigning,emailProtection,timeStamping,OCSPSigning -ext san=dns:cm-c01.mws.mds.xyz,dns:cm-r01nn01.mws.mds.xyz,dns:cm-r01nn02.mws.mds.xyz

Updated jssecerts (in path above) with the new cert as well. Same issue.

I'm running the load balancer and haproxy on the cm-r01nn01/02 . This is a problem since Cloudera opens up ports such as 7180, 7182 and 7183 on all available interfaces. So if I have a VIP running on the same host,
Cloudera services try and do bind to it. Can't really have HAproxy running on port 7183 on the LB VIP if Cloudera services are already bound to the same port.

Tried the Cloudera Manager Hostname Override but only got these errors:

12:25:32.076 AM	WARN	BasicScmProxy	
Exception while getting fetch configDefaults hash: none
java.io.IOException: HTTPS hostname wrong:  should be <srv-c01.mws.mds.xyz>
	at sun.net.www.protocol.https.HttpsClient.checkURLSpoofing(HttpsClient.java:649)
	at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:573)
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1334)
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1309)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:259)
	at com.cloudera.cmf.BasicScmProxy.authenticate(BasicScmProxy.java:277)
	at com.cloudera.cmf.BasicScmProxy.fetch(BasicScmProxy.java:607)
	at com.cloudera.cmf.BasicScmProxy.getFragmentAndHash(BasicScmProxy.java:696)
	at com.cloudera.cmf.DescriptorAndFragments.newDescriptorAndFragments(DescriptorAndFragments.java:65)
	at com.cloudera.cmon.firehose.Main.main(Main.java:396)

Need to spend more time reading on the LB's and CM / CDH.

Thx,
TK