I've enabled Auto-TLS on an existing CDP 7.1.3 cluster as instructed in https://blog.cloudera.com/auto-tls-in-cloudera-data-platform-data-center/ option 2b. Initially after completing the wizard and restarting CM agent and servers agent heartbeat was lost due to missing agent certificates from truststore. I was able to fix this by adding hosts to trustore "Cloudera Manager TLS/SSL Client Trust Store File" /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks with Java Keytool.   However now I'm observing some oddities in services. - YARN, Kafka seem to report in their Role Log at server startup (Error 403 Broken pipe (Write failed) the server declined access to the page or resource.) - If I try to access any logs in the cluster via CM GUI (Clusters -> Service -> Instances -> Any role -> Log files -> Any log -> Download full log ) I get the following message "HTTP ERROR 403 Problem accessing /cmf/process/all/logs/download. Reason: Received fatal alert: unsupported_certificate The server declined access to the page or resource." Other services that I'm observing this is Kafka (connect role) and HDFS (namenode RPC).   Any ideas where should I look at? ... View more

Hi, I've enabled Auto-TLS as instructed in option 2b (https://docs.cloudera.com/cdp-private-cloud-base/7.1.4/security-encrypting-data-in-transit/topics/cm-security-use-case-2.html) but my agents are reporting heartbeat failed in /var/log/cloudera-scm-agent/cloudera-scm-agent.log. Agent at master does not have problems with it's heartbeat. The environment is CDP 7.1.4 with CDP 7.1.3 parcels. I have a development license in place. Also noticed following messages in the certmanager log: "[03/Nov/2020 20:46:58 +0200] 16654 MainThread cert INFO No password file found for host 'masterofanalytics.hemanuniverse.com' at location: /opt/cloudera/AutoTLS/hosts-key-store/masterofanalytics.hemanuniverse.com/cm-auto-host_key.pw. Assuming default in-cluster password." FQDNs and name resolution should be OK. [azureuser@skeletor ~]$ cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 10.0.0.4 masterofanalytics.hemanuniverse.com masterofanalytics 10.0.0.6 skeletor.hemanuniverse.com skeletor 10.0.0.7 hordeprime.hemanuniverse.com hordeprime 10.0.0.8 horlak.hemanuniverse.com horlak Below I've verified the fqdn of all 4 servers in the cluster and verified that their key and certificate matches and owner of certificate. There's an company CA in place that has signed the CSR's. It seems that as agents are having trouble authenticating TLS as I'm running command: "sudo -u cloudera-scm openssl s_client -connect masterofanalytics.hemanuniverse.com:7182 -CAfile /var/lib/cloudera-scm-agent/agent-cert/cm-auto-in_cluster_ca_cert.pem -cert /var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_cert_chain.pem -key /var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_key.pem -servername $(hostname -f) -pass file:/var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_key.pw" this gives me error int STDER "140509671278480:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:s3_pkt.c:1493:SSL alert number 46 140509671278480:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177: CONNECTED(00000003)" [azureuser@masterofanalytics ~]$ for i in `grep 10.0.0 /etc/hosts | awk '{print $2}'`; do ssh $i "python -c 'import socket; print socket.getfqdn(), socket.gethostbyname(socket.getfqdn())'"; ssh $i sudo openssl rsa -noout -modulus -in /var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_key.pem -passin file:/var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_key.pw | openssl md5; ssh $i openssl x509 -noout -modulus -in /var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_cert_chain.pem | openssl md5; ssh $i keytool -printcert -file /var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_cert_chain.pem | grep -e 'Owner:\|Issuer:' | paste -d " " - -; ssh $i sudo -u cloudera-scm openssl s_client -connect masterofanalytics.hemanuniverse.com:7182 -CAfile /var/lib/cloudera-scm-agent/agent-cert/cm-auto-in_cluster_ca_cert.pem -cert /var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_cert_chain.pem -key /var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_key.pem -servername $(hostname -f) -pass file:/var/lib/cloudera-scm-agent/agent-cert/cm-auto-host_key.pw; done masterofanalytics.hemanuniverse.com 10.0.0.4 (stdin)= 1fa4d9fdd951bc5afb3c4f56d99546dd (stdin)= 1fa4d9fdd951bc5afb3c4f56d99546dd Owner: CN=masterofanalytics.hemanuniverse.com, OU=LINUX, O=hemanuniverse.com, L=Palo Alto, ST=California, C=US Issuer: CN=hemanuniverse-Hulk-CA, DC=hemanuniverse, DC=com depth=1 DC = com, DC = hemanuniverse, CN = hemanuniverse-Hulk-CA verify return:1 depth=0 C = US, ST = California, L = Palo Alto, O = hemanuniverse.com, OU = LINUX, CN = masterofanalytics.hemanuniverse.com verify return:1 CONNECTED(00000003) --- Certificate chain 0 s:/C=US/ST=California/L=Palo Alto/O=hemanuniverse.com/OU=LINUX/CN=masterofanalytics.hemanuniverse.com i:/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA 1 s:/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA i:/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA --- Server certificate -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- subject=/C=US/ST=California/L=Palo Alto/O=hemanuniverse.com/OU=LINUX/CN=masterofanalytics.hemanuniverse.com issuer=/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA --- Acceptable client certificate CA names /DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA /C=US/ST=California/L=Palo Alto/O=hemanuniverse.com/OU=LINUX/CN=masterofanalytics.hemanuniverse.com Client Certificate Types: RSA sign, DSA sign, ECDSA sign Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1 Shared Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1 Peer signing digest: SHA512 Server Temp Key: ECDH, P-256, 256 bits --- SSL handshake has read 3170 bytes and written 3150 bytes --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-GCM-SHA384 Session-ID: 5FA4FE04A84481F20A6F71ED898FAC8C659163E89F2D4E7DAFC20C4476D352DF Session-ID-ctx: Master-Key: 2F0994587F48D1CFFA08BF3BD8F751C5DEA990911B72785FA4AF5AF3F5DED70A7CBC73BA45F0F81411973AE3622A3972 Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1604648452 Timeout : 300 (sec) Verify return code: 0 (ok) --- HTTP/1.1 400 Illegal character SPACE=' ' Content-Type: text/html;charset=iso-8859-1 Content-Length: 70 Connection: close Server: Jetty(9.4.14.v20181114) <h1>Bad Message 400</h1><pre>reason: Illegal character SPACE=' '</pre>closed skeletor.hemanuniverse.com 10.0.0.6 (stdin)= 61c35563b5b41fc7e4ac7c4a14dfaf1e (stdin)= 61c35563b5b41fc7e4ac7c4a14dfaf1e Owner: CN=skeletor.hemanuniverse.com, OU=LINUX, O=hemanuniverse.com, L=Palo Alto, ST=California, C=US Issuer: CN=hemanuniverse-Hulk-CA, DC=hemanuniverse, DC=com depth=1 DC = com, DC = hemanuniverse, CN = hemanuniverse-Hulk-CA verify return:1 depth=0 C = US, ST = California, L = Palo Alto, O = hemanuniverse.com, OU = LINUX, CN = masterofanalytics.hemanuniverse.com verify return:1 140509671278480:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:s3_pkt.c:1493:SSL alert number 46 140509671278480:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177: CONNECTED(00000003) --- Certificate chain 0 s:/C=US/ST=California/L=Palo Alto/O=hemanuniverse.com/OU=LINUX/CN=masterofanalytics.hemanuniverse.com i:/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA 1 s:/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA i:/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA --- Server certificate -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- subject=/C=US/ST=California/L=Palo Alto/O=hemanuniverse.com/OU=LINUX/CN=masterofanalytics.hemanuniverse.com issuer=/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA --- Acceptable client certificate CA names /DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA /C=US/ST=California/L=Palo Alto/O=hemanuniverse.com/OU=LINUX/CN=masterofanalytics.hemanuniverse.com Client Certificate Types: RSA sign, DSA sign, ECDSA sign Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1 Shared Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1 Peer signing digest: SHA512 Server Temp Key: ECDH, P-256, 256 bits --- SSL handshake has read 3126 bytes and written 2799 bytes --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-GCM-SHA384 Session-ID: 5FA4FE086ADB544867E55A118D5D86F678FDE2919FE89CEB92D9A173E4FA5C23 Session-ID-ctx: Master-Key: 60C95476C76E930A0CAA84735504E7EA567E94BACE9C1AEC4F49151FF9B4DA2679578E8F2381897D106425AB366C7EBD Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1604648456 Timeout : 300 (sec) Verify return code: 0 (ok) --- hordeprime.hemanuniverse.com 10.0.0.7 (stdin)= 68d65855b5080169c75bd312d397cd16 (stdin)= 68d65855b5080169c75bd312d397cd16 Owner: CN=hordeprime.hemanuniverse.com, OU=LINUX, O=hemanuniverse.com, L=Palo Alto, ST=California, C=US Issuer: CN=hemanuniverse-Hulk-CA, DC=hemanuniverse, DC=com depth=1 DC = com, DC = hemanuniverse, CN = hemanuniverse-Hulk-CA verify return:1 depth=0 C = US, ST = California, L = Palo Alto, O = hemanuniverse.com, OU = LINUX, CN = masterofanalytics.hemanuniverse.com verify return:1 139973905454992:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:s3_pkt.c:1493:SSL alert number 46 139973905454992:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177: CONNECTED(00000003) --- Certificate chain 0 s:/C=US/ST=California/L=Palo Alto/O=hemanuniverse.com/OU=LINUX/CN=masterofanalytics.hemanuniverse.com i:/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA 1 s:/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA i:/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA --- Server certificate -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- subject=/C=US/ST=California/L=Palo Alto/O=hemanuniverse.com/OU=LINUX/CN=masterofanalytics.hemanuniverse.com issuer=/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA --- Acceptable client certificate CA names /DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA /C=US/ST=California/L=Palo Alto/O=hemanuniverse.com/OU=LINUX/CN=masterofanalytics.hemanuniverse.com Client Certificate Types: RSA sign, DSA sign, ECDSA sign Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1 Shared Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1 Peer signing digest: SHA512 Server Temp Key: ECDH, P-256, 256 bits --- SSL handshake has read 3126 bytes and written 2803 bytes --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-GCM-SHA384 Session-ID: 5FA4FE09116871B4C37F4981A8D2E59186C2E219BF67895DC168FBCDF6BC915D Session-ID-ctx: Master-Key: DF3033C1A66881A7C42AF6692A011771C8C472109B2D6184E80DABDB6AC0FF9B4C12DEAAF716DF4643533F63DBA42522 Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1604648457 Timeout : 300 (sec) Verify return code: 0 (ok) --- horlak.hemanuniverse.com 10.0.0.8 (stdin)= 42de4a2a447c9dd5ad4be2f5949c2c0f (stdin)= 42de4a2a447c9dd5ad4be2f5949c2c0f Owner: CN=horlak.hemanuniverse.com, OU=LINUX, O=hemanuniverse.com, L=Palo Alto, ST=California, C=US Issuer: CN=hemanuniverse-Hulk-CA, DC=hemanuniverse, DC=com depth=1 DC = com, DC = hemanuniverse, CN = hemanuniverse-Hulk-CA verify return:1 depth=0 C = US, ST = California, L = Palo Alto, O = hemanuniverse.com, OU = LINUX, CN = masterofanalytics.hemanuniverse.com verify return:1 CONNECTED(00000003) --- Certificate chain 0 s:/C=US/ST=California/L=Palo Alto/O=hemanuniverse.com/OU=LINUX/CN=masterofanalytics.hemanuniverse.com i:/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA 1 s:/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA i:/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA --- Server certificate -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- subject=/C=US/ST=California/L=Palo Alto/O=hemanuniverse.com/OU=LINUX/CN=masterofanalytics.hemanuniverse.com issuer=/DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA --- Acceptable client certificate CA names /DC=com/DC=hemanuniverse/CN=hemanuniverse-Hulk-CA /C=US/ST=California/L=Palo Alto/O=hemanuniverse.com/OU=LINUX/CN=masterofanalytics.hemanuniverse.com Client Certificate Types: RSA sign, DSA sign, ECDSA sign Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1 Shared Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1 Peer signing digest: SHA512 Server Temp Key: ECDH, P-256, 256 bits --- SSL handshake has read 3126 bytes and written 2795 bytes --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-GCM-SHA384 Session-ID: 5FA4FE0B83BEC694F742FA3FE854478CA59F9F27EC738D22961412EC9590404D Session-ID-ctx: Master-Key: 5147A7A68B36E29321783F165644AA5716736FAE8752C3C136E8751F39148974DBF22879AFE66A273ACD2B77F4192F0A Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1604648459 Timeout : 300 (sec) Verify return code: 0 (ok) --- 140231002048400:error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown:s3_pkt.c:1493:SSL alert number 46 140231002048400:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177: ... View more