Hi, I've enabled Auto-TLS as instructed in option 2b (https://docs.cloudera.com/cdp-private-cloud-base/7.1.4/security-encrypting-data-in-transit/topics/cm... my agents are reporting heartbeat failed in /var/log/cloudera-scm-agent/cloudera-scm-agent.log. Agent at master does not have problems with it's heartbeat. The environment is CDP 7.1.4 with CDP 7.1.3 parcels.
I have a development license in place.
Also noticed following messages in the certmanager log:
"[03/Nov/2020 20:46:58 +0200] 16654 MainThread cert INFO No password file found for host 'masterofanalytics.hemanuniverse.com' at location: /opt/cloudera/AutoTLS/hosts-key-store/masterofanalytics.hemanuniverse.com/cm-auto-host_key.pw. Assuming default in-cluster password."
FQDNs and name resolution should be OK.
[azureuser@skeletor ~]$ cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.0.0.4 masterofanalytics.hemanuniverse.com masterofanalytics
10.0.0.6 skeletor.hemanuniverse.com skeletor
10.0.0.7 hordeprime.hemanuniverse.com hordeprime
10.0.0.8 horlak.hemanuniverse.com horlak
Below I've verified the fqdn of all 4 servers in the cluster and verified that their key and certificate matches and owner of certificate. There's an company CA in place that has signed the CSR's. It seems that as agents are having trouble authenticating TLS as I'm running command:
"sudo -u cloudera-scm openssl s_client -connect masterofanalytics.hemanuniverse.com:7182 -CAfile /var/lib/cloudera-scm-agent/agent-cert/cm-auto-in_cluster_ca_cert.pem -cert /var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_cert_chain.pem -key /var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_key.pem -servername $(hostname -f) -pass file:/var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_key.pw"
this gives me error int STDER "140509671278480:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:s3_pkt.c:1493:SSL alert number 46
140509671278480:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:
CONNECTED(00000003)"
[azureuser@masterofanalytics ~]$ for i in `grep 10.0.0 /etc/hosts | awk '{print $2}'`; do ssh $i "python -c 'import socket; print socket.getfqdn(), socket.gethostbyname(socket.getfqdn())'"; ssh $i sudo openssl rsa -noout -modulus -in /var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_key.pem -passin file:/var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_key.pw | openssl md5; ssh $i openssl x509 -noout -modulus -in /var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_cert_chain.pem | openssl md5; ssh $i keytool -printcert -file /var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_cert_chain.pem | grep -e 'Owner:\|Issuer:' | paste -d " " - -; ssh $i sudo -u cloudera-scm openssl s_client -connect masterofanalytics.hemanuniverse.com:7182 -CAfile /var/lib/cloudera-scm-agent/agent-cert/cm-auto-in_cluster_ca_cert.pem -cert /var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_cert_chain.pem -key /var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_key.pem -servername $(hostname -f) -pass file:/var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_key.pw; done
masterofanalytics.hemanuniverse.com 10.0.0.4
(stdin)= 1fa4d9fdd951bc5afb3c4f56d99546dd
(stdin)= 1fa4d9fdd951bc5afb3c4f56d99546dd
Owner: CN=masterofanalytics.hemanuniverse.com, OU=LINUX, O=hemanuniverse.com, L=Palo Alto, ST=California, C=US Issuer: CN=hemanuniverse-Hulk-CA, DC=hemanuniverse, DC=com
depth=1 DC = com, DC = hemanuniverse, CN = hemanuniverse-Hulk-CA
verify return:1
depth=0 C = US, ST = California, L = Palo Alto, O = hemanuniverse.com, OU = LINUX, CN = masterofanalytics.hemanuniverse.com
verify return:1
CONNECTED(00000003)
---
Certificate chain
0 s:/C=US/ST=California/L=Palo Alto/O=hemanuniverse.com/OU=LINUX/CN=masterofanalytics.hemanuniverse.com
i:/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA
1 s:/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA
i:/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA
---
Server certificate
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=Palo Alto/O=hemanuniverse.com/OU=LINUX/CN=masterofanalytics.hemanuniverse.com
issuer=/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA
---
Acceptable client certificate CA names
/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA
/C=US/ST=California/L=Palo Alto/O=hemanuniverse.com/OU=LINUX/CN=masterofanalytics.hemanuniverse.com
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3170 bytes and written 3150 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 5FA4FE04A84481F20A6F71ED898FAC8C659163E89F2D4E7DAFC20C4476D352DF
Session-ID-ctx:
Master-Key: 2F0994587F48D1CFFA08BF3BD8F751C5DEA990911B72785FA4AF5AF3F5DED70A7CBC73BA45F0F81411973AE3622A3972
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1604648452
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
HTTP/1.1 400 Illegal character SPACE=' '
Content-Type: text/html;charset=iso-8859-1
Content-Length: 70
Connection: close
Server: Jetty(9.4.14.v20181114)
<h1>Bad Message 400</h1><pre>reason: Illegal character SPACE=' '</pre>closed
skeletor.hemanuniverse.com 10.0.0.6
(stdin)= 61c35563b5b41fc7e4ac7c4a14dfaf1e
(stdin)= 61c35563b5b41fc7e4ac7c4a14dfaf1e
Owner: CN=skeletor.hemanuniverse.com, OU=LINUX, O=hemanuniverse.com, L=Palo Alto, ST=California, C=US Issuer: CN=hemanuniverse-Hulk-CA, DC=hemanuniverse, DC=com
depth=1 DC = com, DC = hemanuniverse, CN = hemanuniverse-Hulk-CA
verify return:1
depth=0 C = US, ST = California, L = Palo Alto, O = hemanuniverse.com, OU = LINUX, CN = masterofanalytics.hemanuniverse.com
verify return:1
140509671278480:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:s3_pkt.c:1493:SSL alert number 46
140509671278480:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:
CONNECTED(00000003)
---
Certificate chain
0 s:/C=US/ST=California/L=Palo Alto/O=hemanuniverse.com/OU=LINUX/CN=masterofanalytics.hemanuniverse.com
i:/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA
1 s:/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA
i:/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA
---
Server certificate
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=Palo Alto/O=hemanuniverse.com/OU=LINUX/CN=masterofanalytics.hemanuniverse.com
issuer=/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA
---
Acceptable client certificate CA names
/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA
/C=US/ST=California/L=Palo Alto/O=hemanuniverse.com/OU=LINUX/CN=masterofanalytics.hemanuniverse.com
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3126 bytes and written 2799 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 5FA4FE086ADB544867E55A118D5D86F678FDE2919FE89CEB92D9A173E4FA5C23
Session-ID-ctx:
Master-Key: 60C95476C76E930A0CAA84735504E7EA567E94BACE9C1AEC4F49151FF9B4DA2679578E8F2381897D106425AB366C7EBD
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1604648456
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
hordeprime.hemanuniverse.com 10.0.0.7
(stdin)= 68d65855b5080169c75bd312d397cd16
(stdin)= 68d65855b5080169c75bd312d397cd16
Owner: CN=hordeprime.hemanuniverse.com, OU=LINUX, O=hemanuniverse.com, L=Palo Alto, ST=California, C=US Issuer: CN=hemanuniverse-Hulk-CA, DC=hemanuniverse, DC=com
depth=1 DC = com, DC = hemanuniverse, CN = hemanuniverse-Hulk-CA
verify return:1
depth=0 C = US, ST = California, L = Palo Alto, O = hemanuniverse.com, OU = LINUX, CN = masterofanalytics.hemanuniverse.com
verify return:1
139973905454992:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:s3_pkt.c:1493:SSL alert number 46
139973905454992:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:
CONNECTED(00000003)
---
Certificate chain
0 s:/C=US/ST=California/L=Palo Alto/O=hemanuniverse.com/OU=LINUX/CN=masterofanalytics.hemanuniverse.com
i:/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA
1 s:/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA
i:/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA
---
Server certificate
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=Palo Alto/O=hemanuniverse.com/OU=LINUX/CN=masterofanalytics.hemanuniverse.com
issuer=/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA
---
Acceptable client certificate CA names
/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA
/C=US/ST=California/L=Palo Alto/O=hemanuniverse.com/OU=LINUX/CN=masterofanalytics.hemanuniverse.com
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3126 bytes and written 2803 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 5FA4FE09116871B4C37F4981A8D2E59186C2E219BF67895DC168FBCDF6BC915D
Session-ID-ctx:
Master-Key: DF3033C1A66881A7C42AF6692A011771C8C472109B2D6184E80DABDB6AC0FF9B4C12DEAAF716DF4643533F63DBA42522
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1604648457
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
horlak.hemanuniverse.com 10.0.0.8
(stdin)= 42de4a2a447c9dd5ad4be2f5949c2c0f
(stdin)= 42de4a2a447c9dd5ad4be2f5949c2c0f
Owner: CN=horlak.hemanuniverse.com, OU=LINUX, O=hemanuniverse.com, L=Palo Alto, ST=California, C=US Issuer: CN=hemanuniverse-Hulk-CA, DC=hemanuniverse, DC=com
depth=1 DC = com, DC = hemanuniverse, CN = hemanuniverse-Hulk-CA
verify return:1
depth=0 C = US, ST = California, L = Palo Alto, O = hemanuniverse.com, OU = LINUX, CN = masterofanalytics.hemanuniverse.com
verify return:1
CONNECTED(00000003)
---
Certificate chain
0 s:/C=US/ST=California/L=Palo Alto/O=hemanuniverse.com/OU=LINUX/CN=masterofanalytics.hemanuniverse.com
i:/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA
1 s:/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA
i:/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA
---
Server certificate
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=Palo Alto/O=hemanuniverse.com/OU=LINUX/CN=masterofanalytics.hemanuniverse.com
issuer=/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA
---
Acceptable client certificate CA names
/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA
/C=US/ST=California/L=Palo Alto/O=hemanuniverse.com/OU=LINUX/CN=masterofanalytics.hemanuniverse.com
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3126 bytes and written 2795 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 5FA4FE0B83BEC694F742FA3FE854478CA59F9F27EC738D22961412EC9590404D
Session-ID-ctx:
Master-Key: 5147A7A68B36E29321783F165644AA5716736FAE8752C3C136E8751F39148974DBF22879AFE66A273ACD2B77F4192F0A
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1604648459
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
140231002048400:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:s3_pkt.c:1493:SSL alert number 46
140231002048400:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:
