Auto TLS. Cloudera agent unable to send heartbeat

[#1]

Hi,

Im having issue with the auto-tls option and a bit confuse on the setup.

After installing agent, i was unable to get heartbeat from all my agents. Do i need to run the option in all the agents?

[#2]

Hi Dennistanpunya,

The pictures do show some useful information. The agent host is unable to heartbeat in, and CM server setup/installation doesn't appear to have fully completed the Cloudera Manager setup or know about this node.

A step may have been missed while setting up the Cloudera Manager CA. There should be more details in the /var/log/cloudera-scm-agent/certmanager.log if the certmanager run had any issues.

Was Cloudera Manager server started? its required to login to the Cloudera Manager and add the new nodes to Cloudera manager Hosts in the UI (or via API). During the agent installation Cloudera Manager Server will ssh login as root to install the agent rpm's and distribute the proper TLS keys and configurations for each node automatically.

We've also included some links below, specifically to the section on configuring TLS encryption for Cloudera Manager and CDH using Auto-tls. From your description, the section "Enabling Auto-TLS to automatically create certificates" should help.

https://www.cloudera.com/documentation/enterprise/6/6.0/topics/install_cm_server.html

https://www.cloudera.com/documentation/enterprise/latest/topics/auto_tls.html#auto_tls_cm_ca

[#3]

HI Tim,

here is from my /var/log/cloudera-scm-agent/certmanager.log

I did follow evry single step in https://www.cloudera.com/documentation/enterprise/6/latest/topics/install_cm_server.html#install_cm_...

for auto-tls.

but... heartbeat still fails

[#4]

do i need to run this in every host?

JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera /opt/cloudera/cm-agent/bin/certmanager setup --configure-services

firewall is disabled for host and server.

Did i miss anymore steps?

Without that heartbeat, i cant proceed with the installation.

[#5]

i have also have a proper /etc/hosts file;

[root@cl-wor2 ~]# cat /etc/hosts
127.0.0.1 localhost
192.168.44.10 cl-cmu.cloudera.de cl-cmu
192.168.44.11 cl-mas1.cloudera.de cl-mas1
192.168.44.12 cl-mas2.cloudera.de cl-mas2
192.168.44.13 cl-wor1.cloudera.de cl-wor1
192.168.44.14 cl-wor2.cloudera.de cl-wor2
192.168.44.15 cl-wor3.cloudera.de cl-wor3

and configured /etc/cloudera-scm-agent/config.ini to set ;

[Security]
# Use TLS and certificate validation when connecting to the CM server.
use_tls=1

[#6]

i tried this from one of my host;

[root@cl-wor2 ~]# openssl s_client -connect cl-cmu.cloudera.de:7182
CONNECTED(00000003)
depth=1 C = US, ST = CA, CN = SCM Local CA on cl-cmu.cloudera.de
verify error:num=19:self signed certificate in certificate chain
139810545244048:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:s3_pkt.c:1493:SSL alert number 42
139810545244048:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:
---
Certificate chain
0 s:/C=US/ST=CA/CN=cl-cmu.cloudera.de
i:/C=US/ST=CA/CN=SCM Local CA on cl-cmu.cloudera.de
1 s:/C=US/ST=CA/CN=SCM Local CA on cl-cmu.cloudera.de
i:/C=US/ST=CA/CN=SCM Local CA on cl-cmu.cloudera.de
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEnDCCAwSgAwIBAgIBATANBgkqhkiG9w0BAQsFADBHMQswCQYDVQQGEwJVUzEL
MAkGA1UECAwCQ0ExKzApBgNVBAMMIlNDTSBMb2NhbCBDQSBvbiBjbC1jbXUuY2xv
dWRlcmEuZGUwHhcNMTgxMjE4MDUyMTE1WhcNMTkxMjE4MjM1OTU5WjA3MQswCQYD
VQQGEwJVUzELMAkGA1UECAwCQ0ExGzAZBgNVBAMMEmNsLWNtdS5jbG91ZGVyYS5k
ZTCCAaIwDQYJKoZIhvcNAQEBBQADggGPADCCAYoCggGBAMEs3nxmgsHWKWd9+LH3
UZ8h+YU3FlcgfDcGZha02CV8ZuFdACTd4vpySftJXyvA1jB5BM/k/na9dTbJf2G5
Ziba5x3pwb5rrzbuEPDGg0sTQallph4sQTpUPEj9LfBpel95SR87cYmDDwCPLuE7
ghuodVEHPVlUT+lTp7pOoSJIjLC2bId9R9yh0XfTXsp0RDpcKLkRbW4q0Zm6ah+o
EMqbXolymd+H1Lo142ulnp17an/Bbq3YNyng26EN1Kjb0Z1tny6UwPg09GAzwXKq
cpZJEg1mHucZN+am7lJVDGsBEJ291bOAbSsrTaiVhOJFdn6KZHCyjVxDLBhfsjuJ
k86SshmyXOjpyCDtCA40vK0y6fK3g+pdMa1tetzdVf5HFG6sj+pHZVx+6vJJkRlV
CuySmCrXDLkqY8BZdLrIpSx//NqwMJ4+f5nOBIDMWAKEuFnqJe9/mQFHY9ccttGM
Otg6QLQQiWKGL0r2MtzrlkPSE+6fhLGKaEHmLmY6b6AQBQIDAQABo4GiMIGfMAwG
A1UdEwEB/wQCMAAwHQYDVR0OBBYEFLnCRcFACpD22MsYNNjRFUEr+KgOMB8GA1Ud
IwQYMBaAFIqQFuCugOXnMYK0VkoXOHNl0qC/MA4GA1UdDwEB/wQEAwIDqDAgBgNV
HSUBAf8EFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwHQYDVR0RBBYwFIISY2wtY211
LmNsb3VkZXJhLmRlMA0GCSqGSIb3DQEBCwUAA4IBgQDizgO0L50MZDy0TtHXg9XL
eOFAbk3jhHdHpFB03+qh619cq+eCNwg/Zzuv1y1zUinp1jKZh8rhGPlvmkv9Ljy0
GZhh/qIitUCXnQuNTz9plObdBA6ZctPBe4TgDi+rDN5drxDnLZ7zG6piEpP1ErWg
4zYlqwRoSs3Q9Rd+Go7tgoBRx02fJx/QoWa7JigXirt+M3epBl0BwPlogMaStRZM
pOMkZJab9cyFWBV8xqCtPOisFLwmKbPnFFsT5xawdAhNRjnpJzlZ2bhsjHg2q4M3
NfUJRvKDQeo0YJHPUq9J1raSmCTxnpU3J1V9xpzhS9NkhQh6buWjPBz2mli2PxsF
sIp+W+RHWeku8gUO3xVUgN838lhHv6ihOn2O+mpRXwINGp+KOGJpFl6wYJVgzHuS
MqrhJyRyEX148azWb7sgKcTAqdk09llS5+hxh9QnmW0G4nAedmjSmBqPnv/sklal
7K3e95u6f7rX8J8bOFYftpb/YntMuDiKRRVFCnBQkHc=
-----END CERTIFICATE-----
subject=/C=US/ST=CA/CN=cl-cmu.cloudera.de
issuer=/C=US/ST=CA/CN=SCM Local CA on cl-cmu.cloudera.de
---
Acceptable client certificate CA names
/C=US/ST=CA/CN=SCM Local CA on cl-cmu.cloudera.de
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3004 bytes and written 178 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-SHA256
Server public key is 3072 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES128-SHA256
Session-ID: 5C18A20A38BB8B1FF4412C1528AD35D62BF29B3DE729E08E004FC7C49EF5B6C9
Session-ID-ctx:
Master-Key: E9E104DD67ABDD5660220F64659A5FD6DF9CB77BF930C158FB3D13401B842E1270ADA79E8CA7619C4A3FB4277914A257
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1545118218
Timeout : 300 (sec)
Verify return code: 19 (self signed certificate in certificate chain)
---
[root@cl-wor2 ~]#

[#7]

Receiving alert bad certificate (code 42) means the server demands you authenticate with a certificate, and you did not do so, and that caused the handshake failure. A few lines before the line SSL handshake has read ... and written ... you should see a line Acceptable client certificate CA names usually followed by several lines identifying CAs, possibly followed by a line beginning Client Certificate Types and maybe some about Requested Signature Algorithms depending on your OpenSSL version and the negotiated protocol.

From here i understand my host doesnt have any cert given. I thought the auto-tls will handle the granting of certs to agent host?

 

i have enabled auto tls;

[#8]

Hi Tim,

Just discover this in /var/log/cloudera-scm-server/cloudera-scm-server.log

2018-12-18 15:11:34,623 INFO NodeConfiguratorThread-7-3:com.cloudera.server.cmf.node.NodeConfigurator: Executing bash -c 'bash /tmp/scm_prepare_node.I1as8YVq/scm_prepare_node.sh --server_version 6.0.0 --server_build 530873 --packages /tmp/scm_prepare_node.I1as8YVq/packages.scm --always /tmp/scm_prepare_node.I1as8YVq/always_install.scm --x86_64 /tmp/scm_prepare_node.I1as8YVq/x86_64_packages.scm --certtar /tmp/scm_prepare_node.I1as8YVq/cert.tar --unlimitedJCE false --javaInstallStrategy NONE --agentUserMode ROOT --cm http://cl-cmu.cloudera.de/Cloud --skipCloudConfig false -h cl-cmu.cloudera.de | tee /tmp/scm_prepare_node.I1as8YVq/scm_prepare_node.log; exit ${PIPESTATUS[0]}' on cl-wor1.cloudera.de
2018-12-18 15:11:34,625 ERROR NodeConfiguratorThread-7-0:com.cloudera.server.cmf.node.NodeConfigurator: Did not generate Auto-TLS certificates because of missing enterprise license
2018-12-18 15:11:34,625 INFO NodeConfiguratorThread-7-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: cl-cmu.cloudera.de: Transitioning from COPY_FILES (PT0.512S) to CHMOD
2018-12-18 15:11:34,626 INFO NodeConfiguratorThread-7-0:com.cloudera.server.cmf.node.NodeConfigurator: Executing chmod a+x /tmp/scm_prepare_node.cELLGWNq/scm_prepare_node.sh on cl-cmu.cloudera.de
2018-12-18 15:11:34,639 INFO NodeConfiguratorThread-7-0:com.cloudera.server.cmf.node.NodeConfiguratorProgress: cl-cmu.cloudera.de: Transitioning from CHMOD (PT0.014S) to EXECUTE_SCRIPT
2018-12-18 15:11:34,679 INFO

Is this the reason im having TLS issue? 

Can i proceed the installation without TLS enabled?

[#9]

Hi,

Anyone can help me on this matter?

[#10]

@dennistanpunya,

Yes, an enterprise license is required to use auto-tls.  This was introduced in Cloudera Manager 6.1

It appears we may not have made that clear in the documentation.  I'll address this internally.

For now, you can try the feature with the Trial license, but the feature will not function if an express license is in use.