Reply
Explorer
Posts: 16
Registered: ‎11-27-2017

Failed to add host

[ Edited ]

I'm trying to add a host to a single node cluster after configuring it with tls/ssl and kerberos but am unable to.  What should I check?

 

 Installation failed. Failed to receive heartbeat from agent.

  • Ensure that the host's hostname is configured properly.
  • Ensure that port 7182 is accessible on the Cloudera Manager Server (check firewall rules).
  • Ensure that ports 9000 and 9001 are not in use on the host being added.
  • Check agent logs in /var/log/cloudera-scm-agent/ on the host being added. (Some of the logs can be found in the installation details).
  • If Use TLS Encryption for Agents is enabled in Cloudera Manager (Administration -> Settings -> Security), ensure that /etc/cloudera-scm-agent/config.ini has use_tls=1 on the host being added. Restart the corresponding agent and click the Retry link here.

 

>>SSLError: unexpected eof
>>[25/May/2018 11:43:31 +0000] 11441 MainThread agent ERROR Heartbeating to 192.168.0.11:7182 failed.
>>Traceback (most recent call last):
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.14.3-py2.7.egg/cmf/agent.py", line 1424, in _send_heartbeat
>> self.max_cert_depth)
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.14.3-py2.7.egg/cmf/https.py", line 138, in __init__
>> self.conn.connect()
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/M2Crypto-0.24.0-py2.7-linux-x86_64.egg/M2Crypto/httpslib.py", line 59, in connect
>> sock.connect((self.host, self.port))
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/M2Crypto-0.24.0-py2.7-linux-x86_64.egg/M2Crypto/SSL/Connection.py", line 195, in connect
>> ret = self.connect_ssl()
>> File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/M2Crypto-0.24.0-py2.7-linux-x86_64.egg/M2Crypto/SSL/Connection.py", line 188, in connect_ssl
>> return m2.ssl_connect(self.ssl, self._timeout)

>>SSLError: unexpected eof
>>[25/May/2018 11:43:32 +0000] 11441 MainThread agent INFO Stopping agent...
>>[25/May/2018 11:43:32 +0000] 11441 MainThread agent INFO No extant cgroups; unmounting any cgroup roots
>>[25/May/2018 11:43:32 +0000] 11441 MainThread agent INFO 1 processes are being managed; Supervisor will continue to run.
>>[25/May/2018 11:43:32 +0000] 11441 MainThread _cplogging INFO [25/May/2018:11:43:32] ENGINE Bus STOPPING
>>[25/May/2018 11:43:32 +0000] 11441 MainThread _cplogging INFO [25/May/2018:11:43:32] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('phd-node1', 9000)) shut down
>>[25/May/2018 11:43:32 +0000] 11441 MainThread _cplogging INFO [25/May/2018:11:43:32] ENGINE Stopped thread '_TimeoutMonitor'.
>>[25/May/2018 11:43:32 +0000] 11441 MainThread _cplogging INFO [25/May/2018:11:43:32] ENGINE Bus STOPPED
>>[25/May/2018 11:43:32 +0000] 11441 MainThread _cplogging INFO [25/May/2018:11:43:32] ENGINE Bus STOPPING
>>[25/May/2018 11:43:32 +0000] 11441 MainThread _cplogging INFO [25/May/2018:11:43:32] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('phd-node1', 9000)) already shut down
>>[25/May/2018 11:43:32 +0000] 11441 MainThread _cplogging INFO [25/May/2018:11:43:32] ENGINE No thread running for None.
>>[25/May/2018 11:43:32 +0000] 11441 MainThread _cplogging INFO [25/May/2018:11:43:32] ENGINE Bus STOPPED
>>[25/May/2018 11:43:32 +0000] 11441 MainThread _cplogging INFO [25/May/2018:11:43:32] ENGINE Bus EXITING
>>[25/May/2018 11:43:32 +0000] 11441 MainThread _cplogging INFO [25/May/2018:11:43:32] ENGINE Bus EXITED
>>[25/May/2018 11:43:32 +0000] 11441 MainThread agent INFO Agent exiting; caught signal 15
>>[25/May/2018 11:43:32 +0000] 11441 Dummy-13 daemonize WARNING Stopping daemon.
END (0)
end of agent logs.
scm agent started
Installation script completed successfully.

Expert Contributor
Posts: 133
Registered: ‎01-08-2018

Re: Failed to add host

You can check the last of the points you have listed as it is almost sure it will fail to hearbeat, if you have not manually did that step (set use_tls=1 and restart agent)

If Use TLS Encryption for Agents is enabled in Cloudera Manager (Administration -> Settings -> Security), ensure that /etc/cloudera-scm-agent/config.ini has use_tls=1 on the host being added. Restart the corresponding agent and click the Retry link here.

If you did this already, then make sure that the configured keystore and truststore files, have been copied to the new host.

Explorer
Posts: 16
Registered: ‎11-27-2017

Re: Failed to add host

Hi Gekas, thanks for the reply. I copied the config.ini from the head node to the compute node to make sure all the config items are the same.  I've verified use_tls=1.  Anything else that might cause the additional node to not heartbeat?

Highlighted
Expert Contributor
Posts: 133
Registered: ‎01-08-2018

Re: Failed to add host

Have you checked the trust store file that it is copied to the host?
Explorer
Posts: 16
Registered: ‎11-27-2017

Re: Failed to add host

I copied $JAVA_HOME/jre/lib/security/jssecacerts from the head node to the compute node.  What should I check for?  How do I check the trust store?

Explorer
Posts: 18
Registered: ‎03-20-2017

Re: Failed to add host

I am facing exact problem. How did you resolve it?
I would appreciate if you can help me with resolution.
Thanks
Posts: 911
Topics: 1
Kudos: 208
Solutions: 111
Registered: ‎04-22-2014

Re: Failed to add host

@xBigDatax,

 

(1)

 

First,please provide the information you used to assess you are seeing the exact problem.  Stack trace variances can have major implications for how we approach a problem such as this.

 

(2)

 

We need to know what you have set for the following in Cloudera Manager (checked or unchecked):

 

In Cloudera Manager (Administration --> Settings)

- Use TLS Encryption for Agents

- Use TLS Authentication of Agents to Server

 

(3)

 

We need to know what you have configured in the config.ini regarding security on the host that cannot heartbeat:

 

# egrep '(cert|key|tls)' /etc/cloudera-scm-agent/config.ini |grep -v "^#"

 

NEXT STEPS:

 

With the above information, we can determine if your agent and Cloudera Manager security settings align.  Depending on what we find, we may need to take further steps.

 

If you are using the Wizard to add a new host, you will need to manually copy over a config.ini from a "good" host and then create all the security files that are referenced if there are any.

 

 

Explorer
Posts: 16
Registered: ‎11-27-2017

Re: Failed to add host

Tell me about your environment.  I'll try to help as best I can. 

 

Some of the issues I came across while setting up the cluster was using a wildcard cert instead of single certs for each node and I had to setup a DNS server on the head node.

Explorer
Posts: 18
Registered: ‎03-20-2017

Re: Failed to add host

1) below error in both Edge nodes

[19/Sep/2018 19:01:07 +0000] 14599 MainThread agent ERROR Heartbeating to myucbpaabdapp03:7182 failed.
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.14.3-py2.6.egg/cmf/agent.py", line 1424, in _send_heartbeat
self.max_cert_depth)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.14.3-py2.6.egg/cmf/https.py", line 138, in __init__
self.conn.connect()
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/M2Crypto-0.21.1-py2.6-linux-x86_64.egg/M2Crypto/httpslib.py", line 50, in connect
self.sock.connect((self.host, self.port))
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/M2Crypto-0.21.1-py2.6-linux-x86_64.egg/M2Crypto/SSL/Connection.py", line 185, in connect
ret = self.connect_ssl()
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/M2Crypto-0.21.1-py2.6-linux-x86_64.egg/M2Crypto/SSL/Connection.py", line 178, in connect_ssl
return m2.ssl_connect(self.ssl)
SSLError: unexpected eof

 

2)

TLS Authentication of Agents to Server = Checked

Use TLS Encryption for Agents= Checked

 

3) [root@myucbpaabdapp25 security]# egrep '(cert|key|tls)' /etc/cloudera-scm-agent/config.ini |grep -v "^#"
use_tls=1
verify_cert_file=/opt/cloudera/security/x509/agents.pem

 

[root@myucbpaabdapp26 ~]# egrep '(cert|key|tls)' /etc/cloudera-scm-agent/config.ini |grep -v "^#"
use_tls=1
verify_cert_file=/opt/cloudera/security/x509/agents.pem
[root@myucbpaabdapp26 ~]#

 

Explorer
Posts: 18
Registered: ‎03-20-2017

Re: Failed to add host

Please look for self signed certificate details

 

[root@myxxxxxxxxxxxx25 jks]# openssl s_client -connect myxxxxxxxxxxxx03:7182 -CAfile <(keytool -list -rfc -keystore /opt/cloudera/security/jks/cimbbda.truststore < /dev/null) < /dev/null
Enter keystore password:
***************** WARNING WARNING WARNING *****************
* The integrity of the information stored in your keystore *
* has NOT been verified! In order to verify its integrity, *
* you must provide your keystore password. *
***************** WARNING WARNING WARNING *****************

CONNECTED(00000003)
depth=0 C = , ST = , L = , O = , OU = , CN = myxxxxxxxxxxxx03
verify return:1
140385017821000:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:184:
---
Certificate chain
0 s:/C=/ST=/L=/O=/OU=/CN=myxxxxxxxxxxxx03
i:/C=/ST=/L=/O=/OU=/CN=myxxxxxxxxxxxx03
---
Server certificate

Announcements