adding a new node to an existing cluster with TLS/SSL enabled (all three levels) results in an error during parcel activation. The add host wizard goes through the installation (cloudera -scm-agen/config.ini file already configured to use tls and to use server certificate and key, and also the CM certificate there in the variable verify_cert_file)
The agent log contains an ERROR regarding the SSL verification. I dont understand, the cloudera manager certificate is located in that new host. Also the heartbeat is working, so the metrics about CPU/IO are propagated to the CM.
The only workaround I have found is to disable SSL for the cloudera manager and then run the parcel distribution and enable SSL for cloudera manager.
URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)> [06/Oct/2017 15:41:12 +0000] 7072 Thread-13 downloader ERROR Failed fetching torrent: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)> Traceback (most recent call last): File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.11.1-py2.7.egg/cmf/downloader.py", line 263, in download cmf.https.ssl_url_opener.fetch_to_file(torrent_url, torrent_file) File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.11.1-py2.7.egg/cmf/https.py", line 177, in fetch_to_file resp = self.open(req_url) File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.11.1-py2.7.egg/cmf/https.py", line 172, in open return self.opener(*pargs, **kwargs) File "/usr/lib64/python2.7/urllib2.py", line 431, in open response = self._open(req, data) File "/usr/lib64/python2.7/urllib2.py", line 449, in _open '_open', req) File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain result = func(*args) File "/usr/lib64/python2.7/urllib2.py", line 1258, in https_open context=self._context, check_hostname=self._check_hostname) File "/usr/lib64/python2.7/urllib2.py", line 1214, in do_open raise URLError(err) URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)>
# Hostname of the CM server.
# Port that the CM server is listening on.
I am quite sure that I missed something, but dont know what.
At this time, the Add Host Wizard does not support adding a host while TLS is enabled in Cloudera Manager.
The best way to add a new host is as follows:
Install agent/daemon rpms.
yum install cloudera-manager-agent
Edit the /etc/cloudera-scm-agent/config.ini with TLS properties
Ensure the certificate files are in the locations specified.
Start the agent:
service cloudera-scm-agent start
In Cloudera Manager, Go to Hosts -> All Hosts.
Verify that the host is heartbeating to Cloudera Manager. If you see it and it is heartbeating (last heartbeat should be less than 15 seconds ago) then Click "Add New Hosts to Cluster"
In the Wizard, in the "Specify hosts for your CDH cluster installation." page, you should see a tab named "managed hosts" or something like that. Click it.
Select the host (it should appear with a checkbox next to it).
Continue with the wizard.
If you configured your agent for TLS and are still getting the exception regarding CERTIFICATE_VERIFY_FAILED, that usually indicates the agent cannot find trust for the signer of the Cloudera Manager certificate. Let us know if you still see that.
The only workaround is to turn off the TLS for the Cloudera Manager.
Then the cloudera-scm-agent starts to pull the parcel tokens.
I dont know why it does not trust the Cloudera Manager certificate if it is stored and configured properly. If it would not trust the CM certificate then also the Heartbeating would fail. But in my case the heartbeating is working.
This is actually a known bug on Cloudera's side since the parcel download component of the agent has incorrect TLS/SSL logic that is fixed in CM versions 5.12+.
A workaround without disabling TLS:
Add the Root CA of the CM certificate to the /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem.
If you don't have the Root CA of the CM certificate you can obtain it by doing:
openssl s_client -connect CM_HOST:7183 -showcerts
And then copying all of the below sections into a file:
Before adding the contents of this file to /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem.