Reply
Expert Contributor
Posts: 120
Registered: ‎07-01-2015

Parcels distribution in secure CM

Hi,

 adding a new node to an existing cluster with TLS/SSL enabled (all three levels) results in an error during parcel activation. The add host wizard goes through the installation (cloudera -scm-agen/config.ini file already configured to use tls and to use server certificate and key, and also the CM certificate there in the variable verify_cert_file)

 

The agent log contains an ERROR regarding the SSL verification. I dont understand, the cloudera manager certificate is located in that new host. Also the heartbeat is working, so the metrics about CPU/IO are propagated to the CM.

 

The only workaround I have found is to disable SSL for the cloudera manager and then run the parcel distribution and enable SSL for cloudera manager.

 

URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)>
[06/Oct/2017 15:41:12 +0000] 7072 Thread-13 downloader   ERROR    Failed fetching torrent: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)>
Traceback (most recent call last):
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.11.1-py2.7.egg/cmf/downloader.py", line 263, in download
    cmf.https.ssl_url_opener.fetch_to_file(torrent_url, torrent_file)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.11.1-py2.7.egg/cmf/https.py", line 177, in fetch_to_file
    resp = self.open(req_url)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.11.1-py2.7.egg/cmf/https.py", line 172, in open
    return self.opener(*pargs, **kwargs)
  File "/usr/lib64/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib64/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib64/python2.7/urllib2.py", line 1258, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/usr/lib64/python2.7/urllib2.py", line 1214, in do_open
    raise URLError(err)
URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)>

config.ini 

 

[General]
# Hostname of the CM server.
server_host=xxxxx.eu-west-1.compute.internal

# Port that the CM server is listening on.
server_port=7182


max_collection_wait_seconds=10.0

metrics_url_timeout_seconds=30.0

task_metrics_timeout_seconds=5.0

monitored_nodev_filesystem_types=nfs,nfs4,tmpfs

local_filesystem_whitelist=ext2,ext3,ext4,xfs

impala_profile_bundle_max_bytes=1073741824

stacks_log_bundle_max_bytes=1073741824

stacks_log_max_uncompressed_file_size_bytes=5242880

orphan_process_dir_staleness_threshold=5184000

orphan_process_dir_refresh_interval=3600

scm_debug=INFO

dns_resolution_collection_interval_seconds=60

dns_resolution_collection_timeout_seconds=30

[Security]
use_tls=1
max_cert_depth=9
verify_cert_file=/opt/cdh-security/pki/cm.cert
client_key_file=/opt/cdh-security/pki/server.key
client_keypw_file=/etc/cloudera-scm-agent/agentkey.pw
client_cert_file=/opt/cdh-security/pki/server.cert

 

I am quite sure that I missed something, but dont know what.

Any ideas?

Thanks

 

Posts: 437
Topics: 1
Kudos: 102
Solutions: 54
Registered: ‎04-22-2014

Re: Parcels distribution in secure CM

@Tomas79,

 

At this time, the Add Host Wizard does not support adding a host while TLS is enabled in Cloudera Manager.

The best way to add a new host is as follows:

 

(1)

 

Install agent/daemon rpms.

For example:

 

yum install cloudera-manager-agent

 

(2)

 

Edit the /etc/cloudera-scm-agent/config.ini with TLS properties

Ensure the certificate files are in the locations specified.

 

(3)

 

Start the agent:

service cloudera-scm-agent start

 

(4)

 

In Cloudera Manager, Go to Hosts -> All Hosts.

Verify that the host is heartbeating to Cloudera Manager.  If you see it and it is heartbeating (last heartbeat should be less than 15 seconds ago) then Click "Add New Hosts to Cluster"

 

(5)

 

In the Wizard, in the "Specify hosts for your CDH cluster installation." page, you should see a tab named "managed hosts" or something like that.  Click it.

Select the host (it should appear with a checkbox next to it).

Continue with the wizard.

 

----------------

 

If you configured your agent for TLS and are still getting the exception regarding CERTIFICATE_VERIFY_FAILED, that usually indicates the agent cannot find trust for the signer of the Cloudera Manager certificate.  Let us know if you still see that.

Highlighted
Expert Contributor
Posts: 120
Registered: ‎07-01-2015

Re: Parcels distribution in secure CM

The only workaround is to turn off the TLS for the Cloudera Manager.

Then the cloudera-scm-agent starts to pull the parcel tokens. 

 

I dont know why it does not trust the Cloudera Manager certificate if it is stored and configured properly. If it would not trust the CM certificate then also the Heartbeating would fail. But in my case the heartbeating is working.

 

 

 

Cloudera Employee
Posts: 1
Registered: ‎11-17-2017

Re: Parcels distribution in secure CM

This is actually a known bug on Cloudera's side since the parcel download component of the agent has incorrect TLS/SSL logic that is fixed in CM versions 5.12+.

 

A workaround without disabling TLS:

 

Add the Root CA of the CM certificate to the /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem.

 

If you don't have the Root CA of the CM certificate you can obtain it by doing:

 

openssl s_client -connect CM_HOST:7183 -showcerts

 

And then copying all of the below sections into a file:

----BEGIN CERTIFICATE----

 

----END CERTIFICATE----

 

Before adding the contents of this file to /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem.

  

Announcements