Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

SSLError: certificate verify failed

avatar
Explorer

How do I enable further debugging on cloudera-scm-agents?

 

I'm working on deploying the cluster using self signed certificates but I'm running into the below issue and can't get past it:

 

[07/Jul/2019 23:35:05 +0000] 23766 MainThread agent ERROR Heartbeating to cm-r01nn01.mws.mds.xyz:7182 failed.
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeat
self.cfg.max_cert_depth)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__
self.conn.connect()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect
sock.connect((self.host, self.port))
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connect
ret = self.connect_ssl()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_ssl
return m2.ssl_connect(self.ssl, self._timeout)
SSLError: certificate verify failed

What I have in my certificates folder is the following:

 

[root@cm-r01en01 pki]# pwd
/opt/cloudera/security/pki
[root@cm-r01en01 pki]# ls -atlri
total 16
69943167 -rw-r--r--. 1 root root 2385 Apr  1 23:06 cm-r01en01.mws.mds.xyz.keystore.jks
69943152 -rw-r--r--. 1 root root 1453 Apr  1 23:07 cm-r01en01.mws.mds.xyz.pem
 3870062 drwxr-xr-x. 5 root root   37 Apr  1 23:09 ..
69943169 lrwxrwxrwx. 1 root root   62 Apr  1 23:11 server.jks -> /opt/cloudera/security/pki/cm-r01en01.mws.mds.xyz.keystore.jks
69943259 -rw-r--r--. 1 root root 1453 Jul  6 20:01 cm-r01nn01.mws.mds.xyz.pem
69943154 lrwxrwxrwx. 1 root root   53 Jul  6 20:02 rootca.pem -> /opt/cloudera/security/pki/cm-r01nn01.mws.mds.xyz.pem
67689060 lrwxrwxrwx. 1 root root   53 Jul  6 20:36 agent.pem -> /opt/cloudera/security/pki/cm-r01en01.mws.mds.xyz.pem
69943151 drwxr-xr-x. 2 root root 4096 Jul  6 20:36 .
[root@cm-r01en01 pki]#

I'm not 100% sure if I have everything right though.  My cloudera-scm-agent config for that one host:

 

[root@cm-r01en01 pki]# cat /etc/cloudera-scm-agent/config.ini|grep -v "#" | sed -e "/^$/d"
[General]
server_host=cm-r01nn01.mws.mds.xyz
server_port=7182
max_collection_wait_seconds=10.0
metrics_url_timeout_seconds=30.0
task_metrics_timeout_seconds=5.0
monitored_nodev_filesystem_types=nfs,nfs4,tmpfs
local_filesystem_whitelist=ext2,ext3,ext4,xfs
impala_profile_bundle_max_bytes=1073741824
stacks_log_bundle_max_bytes=1073741824
stacks_log_max_uncompressed_file_size_bytes=5242880
orphan_process_dir_staleness_threshold=5184000
orphan_process_dir_refresh_interval=3600
scm_debug=DEBUG
dns_resolution_collection_interval_seconds=60
dns_resolution_collection_timeout_seconds=30
[Security]
use_tls=1
max_cert_depth=9
verify_cert_file=/opt/cloudera/security/pki/agent.pem
verify_cert_dir=/opt/cloudera/security/pki/
[Hadoop]
[Cloudera]
[JDBC]
[Cgroup_Paths]
[root@cm-r01en01 pki]#

cm-r01nn01 is the Name Node.

cm -r01en01 will be the gateway  / entry point to the cluster.  It will also run a few services.  

 

This is CM 6.2 .  I'm looking to go through the certificate process in preparation for a more formal deployment later on w/ official certificates.  Using self signed certs for now for this POC.

 

In particular, what certificate has it tried to load and is looking for?  How do I enable further debug logs to see all the calls it's making and files it's loading?

 

Cheers,
TK

1 ACCEPTED SOLUTION

avatar
Master Guru

@TCloud,

The exception is in the agent and indicates to us that the agent is not able to verify the certificate that was returned by Cloudera Manager during the TLS handshake.

In order to know why, we should look at what host the agent tried to contact (server_host in config.ini) and what certificates were listed in the SAN of the server certificate.

You can use the following command to see what certificate is returned:

openssl s_client -connect $(grep "server_host" /etc/cloudera-scm-agent/config.ini | sed s/server_host=//):7182 </dev/null | openssl x509 -text -noout

Then, check to make sure agent's truststore has the proper certificate that trusts the CM cert.  To test, you can use:

openssl s_client -connect $(grep -v '^#' /etc/cloudera-scm-agent/config.ini | grep "server_host=" | sed s/server_host=//):7182 -CAfile $(grep -v '^#' /etc/cloudera-scm-agent/config.ini | grep "verify_cert_file=" |sed s/verify_cert_file=//) -verify_hostname $(grep -v '^#' /etc/cloudera-scm-agent/config.ini | grep "server_host=" | sed s/server_host=//)</dev/null

 

The above is probably not that elegant, but you should be able to run it as it is.  It will grab your hostname and trust store file from the host's config.ini and then connect to your CM host to do a TLS handshake.  "-verify_hostname" will tell openssl to also do hostname validation to mimic what the agent does.

 

The result code of the above command should give us a better idea of why the handshake is failing.

 

View solution in original post

23 REPLIES 23

avatar
Master Guru

Hi @TCloud ,

 

Thank you for including all this information.

I see that you have both verify_cert_dir and verify_cert_file configured, but one or the other should be chosen.  It appears you are configured for verify_cert_dir based on your directory listing.

 

I would recommend:

 

- commenting out verify_cert_dir by insertinga "#" character at the beginning of the line

- restarting the agent by executing "service cloudera-scm-server restart"

 

If that doesn't work, we can take a closer look at the certificates returned by Cloudera Manager and the certificates in your agent.pem file.

 

 

avatar
Explorer

First, appreciate you looking into this.

 

Just left verify_cert_file in place in /etc/cloudera-scm-agent/config.ini and same issue occurs.

 

Quite certain I got the certs incorrectly configured.  The reason why I'm thinking that is because I pointed agent.pem to various .pem files I have in the above-said folder.  Same result.

 

I also have an HAproxy / Keepalived configuration pointing to a common VIP  between the NN's called cm-c01.mws.mds.xyz .  That works fine for the CM UI, however, I'm trying the agent connection to the primary server directly at first to get that working before trying via the VIP.  So for the purpose of this exercise, I'm not using this VIP.  

 

Here's my cert process:

 

1) Generate a cert.

 

 

keytool -genkeypair -alias $(hostname -f) -keyalg RSA -keysize 2048 -dname "cn=$(hostname -f),OU=MDS,O=MDS,L=Los Angeles,ST=California,C=US" -keypass $(hostname -f) -keystore $(hostname -f).keystore.jks -storepass $(hostname -f) -validity 3650 -ext EKU=serverAuth,clientAuth,codeSigning,emailProtection,timeStamping,OCSPSigning -ext san=dns:$(hostname -f)

 

2) Export cert from keystore.

 

 

keytool -export -alias $(hostname -f) -keystore $(hostname -f).keystore.jks -rfc -file $(hostname -f)-selfsigned.cer

 

3) Import the cert into the java truststores.

 

 

# Import root ca to the JDK Truststore
keytool -import -alias $(hostname -f) -file /opt/cloudera/security/jks/$(hostname -f)-selfsigned.cer -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit



# Import root ca to the JDK Truststore 
keytool -import -alias $(hostname -f) -file /opt/cloudera/security/jks/$(hostname -f)-selfsigned.cer -keystore /usr/java/jdk1.8.0_181-cloudera/jre/lib/security/jssecacerts -storepass changeit

 

 

4) Copy the files around and verify.

 

 

cp -ip /opt/cloudera/security/x509/$(hostname -f).pem /opt/cloudera/security/pki/$(hostname -f).pem;

cp -ip /opt/cloudera/security/x509/$(hostname -f).pem /opt/cloudera/security/pki/;
cp -ip /opt/cloudera/security/jks/$(hostname -f).keystore.jks /opt/cloudera/security/pki;

ln -s /opt/cloudera/security/pki/$(hostname -f).keystore.jks /opt/cloudera/security/pki/server.jks
ln -s /opt/cloudera/security/pki/$(hostname -f).pem /opt/cloudera/security/pki/agent.pem

 

 

openssl x509 -in /opt/cloudera/security/pki/$(hostname -f).pem -noout -text

 

5) Repeat the above on the other hosts.  

 

6) HTTPS / 7183 works fine from CM, just not the agent to server communication, now that I have it enabled.  

 

The parts I'm not clear about stems from my intermediate knowledge of certs:

 

A)  Do I copy the generated cert off the CM / NN server to the other servers in the cluster?

B)  Should I be exporting the private key from the SSL keystore by I) converting to pcs12 II) exporting the cert and key from that?  One of the dozens of options I tried included this step but that didn't work out either when I was trying the connectivity of the cloudera-scm-agent on the NN instead of any other participating host:  

 

keytool -importkeystore \
    -srckeystore cm-r01nn01.mws.mds.xyz.keystore.jks \
    -destkeystore cm-r01nn01.mws.mds.xyz.keystore.p12 \
    -deststoretype PKCS12 \
    -srcalias cm-c01.mws.mds.xyz \
    -deststorepass cm-r01nn01.mws.mds.xyz \
    -destkeypass cm-r01nn01.mws.mds.xyz
	
	
# openssl pkcs12 -in cm-r01nn01.mws.mds.xyz.keystore.p12  -nokeys -out cm-r01nn01.mws.mds.xyz.cert.pem
Enter Import Password:
MAC verified OK
#

# openssl pkcs12 -in cm-r01nn01.mws.mds.xyz.keystore.p12  -nodes -nocerts -out cm-r01nn01.mws.mds.xyz.key.pem
Enter Import Password:
MAC verified OK
#

Thinking I might have the commands right, just not the sequence or I'm missing a step or two.  Further notes and info:

 

[ UTILITY NODE ]

 

[root@cm-r01en01 pki]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts|grep cm
Enter keystore password:  changeit
acraizfnmt-rcm, Mar 26, 2019, trustedCertEntry,
cm-r01nn02.mws.mds.xyz, Apr 14, 2019, trustedCertEntry,
cm-r01en01.mws.mds.xyz, Apr 1, 2019, trustedCertEntry,
cm-r01nn01.mws.mds.xyz, Apr 14, 2019, trustedCertEntry,
[root@cm-r01en01 pki]#

[ Name Node / CM Server]

 

[root@cm-r01nn01 pki]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts|grep cm
Enter keystore password:  changeit
acraizfnmt-rcm, Mar 26, 2019, trustedCertEntry,
cm-r01nn02.mws.mds.xyz, Apr 14, 2019, trustedCertEntry,
cm-c01.mws.mds.xyz, Jul 6, 2019, trustedCertEntry,
cm-r01nn01.mws.mds.xyz, Mar 31, 2019, trustedCertEntry,
[root@cm-r01nn01 pki]#

 

The only other message that I got when moving certs around under the agent.pem symlink, was:

 

SSLError: sslv3 alert bad certificate

 

I'm not clear if that error is a step forward or backwards.  Also, discovered by accident that I may not need to restart the cloudera-scm-agent under every scenario.  Just changing where the symlink agent.pem points too appear to generate different messages, even when the agent is running.  


Thx,
TK

 

avatar
Master Guru

@TCloud,

 

If you are using self-signed certificates, then it sounds as if you may have things right.

For the heartbeat issue, though, we can use openssl to see if we have things configured correctly and also see what public certificate is returned by Cloudera Manager for the heartbeat

 

Run:

 

# openssl s_client -connect cm-r01nn01.mws.mds.xyz:7182 -showcerts -CAfile /opt/cloudera/security/pki/cm-r01en01.mws.mds.xyz.pem

 

I believe you will still get an error now that you have explained some of the backstory.

 

When the agent is making a connection to CM to perform the heartbeat, it needs to do a TLS handshake per your configuration.  Part of this handshake is the client (the agent) deciding if it trusts the certificate returned from CM.

 

In your config.ini, you are connecting to the CM host: cm-r01nn01.mws.mds.xyz

However, your pem file name indicates that you are pointing verify_cert_file to the agent's host's certificate, not the certificate of the Cloudera Manager host.

 

In essence, the agent needs to trust the certificate returned by CM in the handshake.

So, the verify_cert_file file will need to contain the certificate for cm-r01nn01.mws.mds.xyz

 

If that doesn't make sense, provide the output from the openssl command above and then also the output from:

 

# openssl x509 -in /opt/cloudera/security/pki/cm-r01en01.mws.mds.xyz.pem -text -noout

 

From that we can provide further advice.

avatar
Master Guru

NOTE:

 

If you are trying to do agent authentication to CM, then you will need to configure the key/cert for the agent to present to CM.  This is a separate, additional configuration than we have been discussing if you have configured Agent Authentication to CM in Cloudera Manager.

avatar
Explorer

Just briefly:

 

1) 

[root@cm-r01en01 pki]# openssl s_client -connect cm-r01nn01.mws.mds.xyz:7182 -showcerts -CAfile /opt/cloudera/security/pki/cm-r01en01.mws.mds.xyz.pem
CONNECTED(00000003)
depth=0 C = US, ST = California, L = Los Angeles, O = MDS, OU = MDS, CN = cm-r01nn01.mws.mds.xyz
verify return:1
140043391174544:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:s3_pkt.c:1493:SSL alert number 42
140043391174544:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:
---
Certificate chain
 0 s:/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-r01nn01.mws.mds.xyz
   i:/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-r01nn01.mws.mds.xyz
-----BEGIN CERTIFICATE-----
MIID9TCCAt2gAwIBAgIEVUJkFTANBgkqhkiG9w0BAQsFADB1MQswCQYDVQQGEwJV
UzETMBEGA1UECBMKQ2FsaWZvcm5pYTEUMBIGA1UEBxMLTG9zIEFuZ2VsZXMxDDAK
BgNVBAoTA01EUzEMMAoGA1UECxMDTURTMR8wHQYDVQQDExZjbS1yMDFubjAxLm13
cy5tZHMueHl6MB4XDTE5MDQwMTAyMzc1MFoXDTI5MDMyOTAyMzc1MFowdTELMAkG
A1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWExFDASBgNVBAcTC0xvcyBBbmdl
bGVzMQwwCgYDVQQKEwNNRFMxDDAKBgNVBAsTA01EUzEfMB0GA1UEAxMWY20tcjAx
bm4wMS5td3MubWRzLnh5ejCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AIVFcUGgk3fKC1zuKuSTmc6HQLIxvI9fZXjWMOwYagoJjAgKEFUKtjUTgY15IqT9
VXeBuOICTzDvXQbHEh87KAVLoyQa+DkBR9yoyj5hhRUz+fG7erVgvf1AyB5V289Q
ZZir70nzrnRfGdGh4gFCPH2CKvTBndvLYIbDlSh8P5gjLyck7KMW7GrbtZfxcTGD
qBJYB8okK6/Fs6fYD5UggfYIVhEgAorciHCYlnsJagSda/8Mn5a9WtYLjMJY8hEW
RjZxLLI6aunrd/J24ulUEURaWiqjtfna7Q++mDL4GLtf/sUe4xYjt5zy7qMWPh6Q
aGNVcR+bMVgs+e2qATPbiLkCAwEAAaOBjDCBiTBFBgNVHSUEPjA8BggrBgEFBQcD
AQYIKwYBBQUHAwIGCCsGAQUFBwMDBggrBgEFBQcDBAYIKwYBBQUHAwgGCCsGAQUF
BwMJMCEGA1UdEQQaMBiCFmNtLXIwMW5uMDEubXdzLm1kcy54eXowHQYDVR0OBBYE
FCE0pxrJ9cnfzdjidWWTm5YQFoZ/MA0GCSqGSIb3DQEBCwUAA4IBAQBe9m5Tg76b
Uux3yZudIAtISKwSGP2hz4kYpGEM5ykjiY4UAtScFHLcPEJxFN6K6KtZDy02nStI
aLE9dNB3SayLspC6nC1gAB05D8viFKniOVSG8TQgTytOHG/A7UWz5yQF4uJXiETe
de82xiMt75O1jaePFrsMG2twSZxRTLpd52WGit4A5B8LM2ADTR/wsFvQMhvZA+y5
N+1/pw8bQK4SGGWZ4DWXP+Q5bC/6xP5gN1H9bmvvZILNVma7w+ko7Wr8qIfRB8RU
2j1EiHAp99CjA1on1XVH//3TAhRhvXojh+YnZAwe4JCkCI6vVEYSkbgWBK1txYEa
uZw5Rx5pl+gE
-----END CERTIFICATE-----
---
Server certificate
subject=/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-r01nn01.mws.mds.xyz
issuer=/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-r01nn01.mws.mds.xyz
---
Acceptable client certificate CA names
/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-r01nn01.mws.mds.xyz
/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-c01.mws.mds.xyz
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1733 bytes and written 138 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 5D24C76A5F77382AA772B71FCEC240F1DB45EB8FF77E4B930D58C3CBBABE5682
    Session-ID-ctx:
    Master-Key: 20625D24BF6F4044D4E85E71FC57D350607611BD6F6EF42AA643208DBEB8F9CCFF8FB90B43512A0D548E5E0A385C2B3E
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1562691434
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
[root@cm-r01en01 pki]#

 

2) 

[root@cm-r01en01 pki]# openssl x509 -in /opt/cloudera/security/pki/cm-r01en01.mws.mds.xyz.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 155294363 (0x9419a9b)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, ST=California, L=Los Angeles, O=MDS, OU=MDS, CN=cm-r01en01.mws.mds.xyz
        Validity
            Not Before: Apr  2 03:06:32 2019 GMT
            Not After : Mar 30 03:06:32 2029 GMT
        Subject: C=US, ST=California, L=Los Angeles, O=MDS, OU=MDS, CN=cm-r01en01.mws.mds.xyz
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:a8:d6:e1:0d:25:38:62:1c:fc:26:2c:2d:d4:57:
                    f5:d9:54:f1:ee:8b:5d:45:d9:74:86:3d:08:b3:67:
                    4b:63:ef:7e:ae:f0:ae:84:92:1e:45:83:d8:54:f5:
                    0d:cf:ba:22:80:ed:96:30:60:37:d5:82:11:ed:8e:
                    9b:42:4b:81:1d:c1:5b:45:62:00:0d:20:d5:8b:51:
                    18:df:3f:2a:5a:33:dc:c5:85:63:18:ed:b8:0c:58:
                    a7:26:22:61:b3:16:51:c8:97:42:eb:10:0c:67:00:
                    d1:3c:24:eb:2d:08:7c:fb:91:57:d1:1c:8e:3b:81:
                    a9:e6:f4:6c:ff:1a:f6:7d:9a:07:a4:f1:47:50:ac:
                    44:16:d8:27:17:b2:02:2b:eb:4a:ef:1b:34:69:c2:
                    15:a6:92:a4:7b:0a:f0:c0:43:95:91:c0:f5:40:3e:
                    c6:6f:b2:db:e0:5f:9f:ed:10:36:26:db:d3:e1:d5:
                    42:48:19:73:8a:72:7e:2f:f8:92:89:de:1b:42:64:
                    c9:fa:80:a5:38:ad:c9:4f:e8:96:74:38:d5:58:2d:
                    14:31:36:51:22:14:fc:31:a2:74:3e:91:a4:b3:d5:
                    51:de:70:80:42:0f:0a:58:8b:11:e3:89:e3:a9:78:
                    4f:2e:9d:ae:ab:17:2f:fd:ff:9b:ee:a5:d9:d1:6f:
                    b6:71
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication, Code Signing, E-mail Protection, Time Stamping, OCSP Signing
            X509v3 Subject Alternative Name:
                DNS:cm-r01en01.mws.mds.xyz
            X509v3 Subject Key Identifier:
                41:99:DC:C2:1A:EC:17:F5:89:8B:1A:57:38:FE:FF:8B:28:28:41:F6
    Signature Algorithm: sha256WithRSAEncryption
         57:10:c8:55:05:f9:4b:a1:11:3b:90:19:7a:7b:b5:14:e7:f2:
         7c:d4:c9:c4:cc:49:27:4d:e0:28:d6:0d:f6:08:00:cf:45:c4:
         02:19:bc:2a:9c:bf:df:9c:44:db:00:25:51:af:0b:2d:ba:e9:
         68:eb:dd:c9:de:4a:0f:c9:62:7d:c2:b9:24:e9:1a:ce:bb:3d:
         53:11:9d:d7:6c:25:4c:10:b5:36:79:01:81:8f:7b:29:d4:bc:
         d0:60:7a:81:e9:ff:85:67:f2:62:b8:5c:29:fe:6c:68:72:e7:
         c7:8d:4e:eb:91:17:b1:c7:35:df:cd:cf:33:70:a5:55:4b:fc:
         0d:f1:45:d4:0e:b7:7a:03:53:8a:ae:50:3b:43:9b:04:7e:f0:
         ea:2a:da:7d:03:b7:5c:ce:63:44:c4:4c:17:41:02:cd:87:3e:
         19:a4:9b:4c:18:54:b3:3e:39:53:a2:33:fb:94:ad:e4:1c:86:
         4e:48:aa:85:86:ae:b6:5f:8c:73:81:26:47:a8:19:c8:9e:19:
         6d:0a:8d:70:68:f8:c2:26:c8:66:5b:80:c7:57:12:e7:f8:cb:
         78:3c:75:f1:d5:40:2b:64:87:23:9b:82:5a:70:5b:ed:a3:e1:
         78:43:5b:c7:59:72:aa:8b:a6:bc:c8:72:80:3d:22:85:94:3d:
         03:0d:c8:26
[root@cm-r01en01 pki]#

Will reread your post again in a few minutes.

avatar
Explorer

At one point I was receiving this error as well (which made sense to me):

http://gagravarr.org/writing/openssl-certs/errors.shtml

 

However, it did not help out with immediate issue above.

avatar
Explorer

Pointed agent.pem as follows:

 

 

[root@cm-r01en01 pki]# ls -altri
total 36
69943257 -rw-r--r--. 1 cloudera-scm cloudera-scm 2385 Apr  1 23:06 cm-r01en01.mws.mds.xyz.keystore.jks-backup
69943152 -rw-r--r--. 1 cloudera-scm cloudera-scm 1453 Apr  1 23:07 cm-r01en01.mws.mds.xyz.pem
 3870062 drwxr-xr-x. 5 root         root           37 Apr  1 23:09 ..
69943169 lrwxrwxrwx. 1 root         root           62 Apr  1 23:11 server.jks -> /opt/cloudera/security/pki/cm-r01en01.mws.mds.xyz.keystore.jks
69943259 -rw-r--r--. 1 cloudera-scm cloudera-scm 1453 Jul  6 20:01 cm-r01nn01.mws.mds.xyz.pem
69943154 lrwxrwxrwx. 1 root         root           53 Jul  6 20:02 rootca.pem -> /opt/cloudera/security/pki/cm-r01nn01.mws.mds.xyz.pem
69943167 -rw-r--r--. 1 cloudera-scm cloudera-scm 3449 Jul  8 03:32 cm-r01en01.mws.mds.xyz.keystore.jks
71202464 -rw-r--r--. 1 cloudera-scm cloudera-scm 1415 Jul  8 03:52 cm-r01nn01.mws.mds.xyz.cert.pem
71202517 -rw-r--r--. 1 cloudera-scm cloudera-scm 1859 Jul  8 03:53 cm-r01nn01.mws.mds.xyz.key.pem
71305734 -rw-r--r--. 1 cloudera-scm cloudera-scm 3119 Jul  8 03:56 cm-r01nn01.mws.mds.xyz.cert.key.pem
67257529 -rw-r--r--. 1 cloudera-scm cloudera-scm   23 Jul  8 03:57 agent.pw
67689060 lrwxrwxrwx. 1 root         root           53 Jul  9 20:21 agent.pem -> /opt/cloudera/security/pki/cm-r01nn01.mws.mds.xyz.pem
69943151 drwxr-xr-x. 2 root         root         4096 Jul  9 20:21 .
[root@cm-r01en01 pki]# pwd
/opt/cloudera/security/pki
[root@cm-r01en01 pki]#

 

 

Now the error is:

 

 

[09/Jul/2019 20:22:46 +0000] 19927 MainThread agent        ERROR    Heartbeating to cm-r01nn01.mws.mds.xyz:7182 failed.
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeat
    self.cfg.max_cert_depth)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__
    self.conn.connect()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect
    sock.connect((self.host, self.port))
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connect
    ret = self.connect_ssl()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_ssl
    return m2.ssl_connect(self.ssl, self._timeout)
SSLError: sslv3 alert bad certificate
^C
[root@cm-r01en01 pki]#

 

 

Now I tried to extract the key/cert from the NN generated certs like this (convert to pkcs12 then extract the key/cert):

 

 

[root@cm-r01nn01 pki]# ls -altri
total 32
135962830 -rw-r--r--. 1 root         root         1453 Mar 31 22:41 cm-r01nn01.mws.mds.xyz.pem
201424378 drwxr-xr-x. 5 root         root           37 Mar 31 23:42 ..
135962831 lrwxrwxrwx. 1 root         root           53 Mar 31 23:44 agent.pem -> /opt/cloudera/security/pki/cm-r01nn01.mws.mds.xyz.pem
135962833 lrwxrwxrwx. 1 root         root           62 Mar 31 23:45 server.jks -> /opt/cloudera/security/pki/cm-r01nn01.mws.mds.xyz.keystore.jks
135962832 -rw-r--r--. 1 cloudera-scm cloudera-scm 4723 Jul  6 00:00 cm-r01nn01.mws.mds.xyz.keystore.jks
135283830 -rw-r--r--. 1 root         root         1435 Jul  6 00:10 cm-c01.mws.mds.xyz.pem
135044185 -rw-r--r--. 1 root         root         2751 Jul  8 03:49 cm-r01nn01.mws.mds.xyz.keystore.p12
135044187 -rw-r--r--. 1 root         root         1691 Jul  8 03:50 cm-r01nn01.mws.mds.xyz.cert.pem
135962829 drwxr-xr-x. 2 root         root         4096 Jul  8 03:50 .
135044188 -rw-r--r--. 1 root         root         1859 Jul  8 03:50 cm-r01nn01.mws.mds.xyz.key.pem
[root@cm-r01nn01 pki]# pwd
/opt/cloudera/security/pki
[root@cm-r01nn01 pki]#

Then copied them above to the cm-r01en01.mws.mds.xyz client (as you probably noticed).  Now I'm not clear on my next step.  If I read you correctly, seems as if I need the client's key/cert instead?

 

 

Verifying using openssl yields the same error:

 

 

[root@cm-r01en01 pki]# openssl s_client -connect cm-r01nn01.mws.mds.xyz:7182 < cm-r01nn01.mws.mds.xyz.pem
CONNECTED(00000003)
depth=0 C = US, ST = California, L = Los Angeles, O = MDS, OU = MDS, CN = cm-r01nn01.mws.mds.xyz
verify return:1
139746469861264:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:s3_pkt.c:1493:SSL alert number 42
139746469861264:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:
---
Certificate chain
 0 s:/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-r01nn01.mws.mds.xyz
   i:/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-r01nn01.mws.mds.xyz
---
Server certificate
-----BEGIN CERTIFICATE-----
MIID9TCCAt2gAwIBAgIEVUJkFTANBgkqhkiG9w0BAQsFADB1MQswCQYDVQQGEwJV
UzETMBEGA1UECBMKQ2FsaWZvcm5pYTEUMBIGA1UEBxMLTG9zIEFuZ2VsZXMxDDAK
BgNVBAoTA01EUzEMMAoGA1UECxMDTURTMR8wHQYDVQQDExZjbS1yMDFubjAxLm13
cy5tZHMueHl6MB4XDTE5MDQwMTAyMzc1MFoXDTI5MDMyOTAyMzc1MFowdTELMAkG
A1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWExFDASBgNVBAcTC0xvcyBBbmdl
bGVzMQwwCgYDVQQKEwNNRFMxDDAKBgNVBAsTA01EUzEfMB0GA1UEAxMWY20tcjAx
bm4wMS5td3MubWRzLnh5ejCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AIVFcUGgk3fKC1zuKuSTmc6HQLIxvI9fZXjWMOwYagoJjAgKEFUKtjUTgY15IqT9
VXeBuOICTzDvXQbHEh87KAVLoyQa+DkBR9yoyj5hhRUz+fG7erVgvf1AyB5V289Q
ZZir70nzrnRfGdGh4gFCPH2CKvTBndvLYIbDlSh8P5gjLyck7KMW7GrbtZfxcTGD
qBJYB8okK6/Fs6fYD5UggfYIVhEgAorciHCYlnsJagSda/8Mn5a9WtYLjMJY8hEW
RjZxLLI6aunrd/J24ulUEURaWiqjtfna7Q++mDL4GLtf/sUe4xYjt5zy7qMWPh6Q
aGNVcR+bMVgs+e2qATPbiLkCAwEAAaOBjDCBiTBFBgNVHSUEPjA8BggrBgEFBQcD
AQYIKwYBBQUHAwIGCCsGAQUFBwMDBggrBgEFBQcDBAYIKwYBBQUHAwgGCCsGAQUF
BwMJMCEGA1UdEQQaMBiCFmNtLXIwMW5uMDEubXdzLm1kcy54eXowHQYDVR0OBBYE
FCE0pxrJ9cnfzdjidWWTm5YQFoZ/MA0GCSqGSIb3DQEBCwUAA4IBAQBe9m5Tg76b
Uux3yZudIAtISKwSGP2hz4kYpGEM5ykjiY4UAtScFHLcPEJxFN6K6KtZDy02nStI
aLE9dNB3SayLspC6nC1gAB05D8viFKniOVSG8TQgTytOHG/A7UWz5yQF4uJXiETe
de82xiMt75O1jaePFrsMG2twSZxRTLpd52WGit4A5B8LM2ADTR/wsFvQMhvZA+y5
N+1/pw8bQK4SGGWZ4DWXP+Q5bC/6xP5gN1H9bmvvZILNVma7w+ko7Wr8qIfRB8RU
2j1EiHAp99CjA1on1XVH//3TAhRhvXojh+YnZAwe4JCkCI6vVEYSkbgWBK1txYEa
uZw5Rx5pl+gE
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-r01nn01.mws.mds.xyz
issuer=/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-r01nn01.mws.mds.xyz
---
Acceptable client certificate CA names
/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-r01nn01.mws.mds.xyz
/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-c01.mws.mds.xyz
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1733 bytes and written 138 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 5D25323127BD39A3A3E9C85D12EA95BB46D7EF0D1A2D5EBCD68D502DC140431F
    Session-ID-ctx:
    Master-Key: 96673D16341D315F828D023D34D65937D8BCBA58437F9926B74450998EF22C3CBC038BD099F815CFC6053F4630490362
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1562718769
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
[root@cm-r01en01 pki]#

 

 

Admin -> Settings:

 

Use TLS Encryption for Admin Console (Enabled)

Use TLS Encryption for Agents (Enabled)
Use TLS Authentication of Agents to Server (Enabled)

Verify Agent Hostname Against Certificate (Enabled)

 

Cloudera Manager TLS/SSL Server JKS Keystore File Location = /opt/cloudera/security/pki/server.jks
Cloudera Manager TLS/SSL Server JKS Keystore File Password = <PASS = cm-r01nn01.mws.mds.xyz>

 

Supported SSL/TLS versions = SSLv2Hello, and TLSv1.2

 

CMS -> Configuration

TLS/SSL Client Truststore File Location = /etc/pki/ca-trust/extracted/java/jssecacerts

Cloudera Manager Server TLS/SSL Client Trust Store Password = <PASS = changeit >

 

Options 2-3 above are the two that I don't fully grasp yet.  

 

Cheers,
TK

avatar
Master Guru

@TCloud,

 

First, some background on what the settings do:

 

(1)

 

Use TLS Encryption for Agents - This means that Cloudera Manager's heartbeat port (7182) will require TLS.  All agents need in order to use tls is the following:

 

[Security]
use_tls=1

 

(2)

 

If you then want the agents to verify the Cloudera Manager certificate, you can then add the verify_cert_file configuration:

 

[Security]
use_tls=1

verify_cert_file=/path/to/truststore

 

The "verify_cert_file" value is a path to a file that contains one or more certfificates trusted by the agent.  If you are using self-signed certificates, you can simply use the PEM format of the CM server's certificate.  In your case, the file needs to contain the following:

 

-----BEGIN CERTIFICATE-----
MIID9TCCAt2gAwIBAgIEVUJkFTANBgkqhkiG9w0BAQsFADB1MQswCQYDVQQGEwJV
UzETMBEGA1UECBMKQ2FsaWZvcm5pYTEUMBIGA1UEBxMLTG9zIEFuZ2VsZXMxDDAK
BgNVBAoTA01EUzEMMAoGA1UECxMDTURTMR8wHQYDVQQDExZjbS1yMDFubjAxLm13
cy5tZHMueHl6MB4XDTE5MDQwMTAyMzc1MFoXDTI5MDMyOTAyMzc1MFowdTELMAkG
A1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWExFDASBgNVBAcTC0xvcyBBbmdl
bGVzMQwwCgYDVQQKEwNNRFMxDDAKBgNVBAsTA01EUzEfMB0GA1UEAxMWY20tcjAx
bm4wMS5td3MubWRzLnh5ejCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AIVFcUGgk3fKC1zuKuSTmc6HQLIxvI9fZXjWMOwYagoJjAgKEFUKtjUTgY15IqT9
VXeBuOICTzDvXQbHEh87KAVLoyQa+DkBR9yoyj5hhRUz+fG7erVgvf1AyB5V289Q
ZZir70nzrnRfGdGh4gFCPH2CKvTBndvLYIbDlSh8P5gjLyck7KMW7GrbtZfxcTGD
qBJYB8okK6/Fs6fYD5UggfYIVhEgAorciHCYlnsJagSda/8Mn5a9WtYLjMJY8hEW
RjZxLLI6aunrd/J24ulUEURaWiqjtfna7Q++mDL4GLtf/sUe4xYjt5zy7qMWPh6Q
aGNVcR+bMVgs+e2qATPbiLkCAwEAAaOBjDCBiTBFBgNVHSUEPjA8BggrBgEFBQcD
AQYIKwYBBQUHAwIGCCsGAQUFBwMDBggrBgEFBQcDBAYIKwYBBQUHAwgGCCsGAQUF
BwMJMCEGA1UdEQQaMBiCFmNtLXIwMW5uMDEubXdzLm1kcy54eXowHQYDVR0OBBYE
FCE0pxrJ9cnfzdjidWWTm5YQFoZ/MA0GCSqGSIb3DQEBCwUAA4IBAQBe9m5Tg76b
Uux3yZudIAtISKwSGP2hz4kYpGEM5ykjiY4UAtScFHLcPEJxFN6K6KtZDy02nStI
aLE9dNB3SayLspC6nC1gAB05D8viFKniOVSG8TQgTytOHG/A7UWz5yQF4uJXiETe
de82xiMt75O1jaePFrsMG2twSZxRTLpd52WGit4A5B8LM2ADTR/wsFvQMhvZA+y5
N+1/pw8bQK4SGGWZ4DWXP+Q5bC/6xP5gN1H9bmvvZILNVma7w+ko7Wr8qIfRB8RU
2j1EiHAp99CjA1on1XVH//3TAhRhvXojh+YnZAwe4JCkCI6vVEYSkbgWBK1txYEa
uZw5Rx5pl+gE
-----END CERTIFICATE-----

 

(3)

 

Use TLS Authentication of Agents to Server - This means that Cloudera Manager requires that any agent heartbeating will supply its public certificate as means of authenticating to Cloudera Manager.

 

Agent requirements:

Along with the previous configuration, you need the following:

client_key_file=/path/to/private/key/file
client_keypw_file=/path/to/file/containing/password/for/private/key/file
client_cert_file=/path/to/server/certificate/paired/with/agent/private/key

 

For example:

 

[Security]
use_tls=1
max_cert_depth=9
verify_cert_file=/opt/cloudera/security/cacerts/ClouderaSEC_combined.pem
client_key_file=/opt/cloudera/security/x509/host.key
client_keypw_file=/opt/cloudera/security/agent_key_pw
client_cert_file=/opt/cloudera/security/x509/host.pem

 

Cloudera Manager requirement:

CM truststore must contain the signer of each agent's certificate

CM must be configured with a trust store in "Cloudera Manager TLS/SSL Client Trust Store File"

It appears this file may contain all the hosts' agent certificates: 

 

/etc/pki/ca-trust/extracted/java/jssecacerts

 

NEXT STEPS:

 

It appears that your agents are not configured to perform agent authentication so Cloudera Manager is returning the Bad Certificate TLS Alert and failing the TLS Handshake.  This is a good sign as it means the agent seems to now be able to trust the signer of your CM certificate.

 

To correct the situation, you need to make sure that you configure client_key_file, client_keypw_file, and client_cert_file

 

Make sure to restart the agent after adding them.

avatar
Explorer

Got steps 1 and 2 working.  Tyvm.

 

For step 3, I'm working on it now in combination with your post above and noticing this page might not be correct:

 

https://www.cloudera.com/documentation/enterprise/latest/topics/how_to_configure_cm_tls.html#concept...

 

"Cloudera Manager TLS/SSL Certificate Trust Store File" doesn't appear under Administration -> Settings.

Thx,
TK