Cloudera 6 Heartbeat - SSLError: sslv3 alert bad certificate

[#1]

I'm attempting to install Cloudera 6 as a single-node on Centos 7 using VMWare Workstation 12 for learning and POC purposes.

However, I'm running into heartbeat issues.  Initially I get an error saying...

[30/Sep/2018 21:55:05 +0000] 24667 MainThread agent ERROR Heartbeating to cloudera6.localdomain:7182 failed.
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1371, in _send_heartbeat
response = self.requestor.request('heartbeat', heartbeat_data)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 141, in request
return self.issue_request(call_request, message_name, request_datum)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 254, in issue_request
call_response = self.transceiver.transceive(call_request)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 483, in transceive
result = self.read_framed_message()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 489, in read_framed_message
framed_message = response_reader.read_framed_message()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 417, in read_framed_message
raise ConnectionClosedException("Reader read 0 bytes.")
ConnectionClosedException: Reader read 0 bytes.

The install program said...

If Use TLS Encryption for Agents is enabled in Cloudera Manager (Administration -> Settings -> Security), ensure that /etc/cloudera-scm-agent/config.ini has use_tls=1 on the host being added.

I updated the config.ini file, now I'm getting a slightly different error...

[01/Oct/2018 00:23:22 +0000] 27314 MainThread agent ERROR Heartbeating to cloudera6.localdomain:7182 failed.
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1362, in _send_heartbeat
self.cfg.max_cert_depth)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__
self.conn.connect()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 80, in connect
sock.connect((self.host, self.port))
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 304, in connect
ret = self.connect_ssl()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 291, in connect_ssl
return m2.ssl_connect(self.ssl, self._timeout)
SSLError: sslv3 alert bad certificate

The message just repeats continuously.  The SSLError seems to be a new error compared to the original error.

I've tried this a few times, and figured I'd try to match my configuration to what I see in the QuickStart VMs.  My Hosts files is:

127.0.0.1 cloudera6.localdomain cloudera6 localhost localhost.localdomain
::1 localhost localhost.localdomain

The hostname file is :

cloudera6.localdomain

Any recommendations on what I should do next?

Thanks

W.D.

[#2]

@Meister1867,

My first guess is that you accidentally enabled Agent Authentication which means that the Cloudera Manager Server requires certificate authentication but you have not configured the agent for that yet.

Check in Cloudera Manager's Administration --> Settings --> Security section to see if you have Use TLS Authentication of Agents to Server enabled.  If so:

- uncheck Use TLS Authentication of Agents to Server

- restart Cloudera Manager with "service cloudera-scm-server restart"

- verify in the Hosts --> All hosts section that the last heartbeat for your host happened less than 15 seconds ago.

If so, that was likely the cause.

If you wish to enable Agent Authentication to CM, then you can use the following section to guide you:

https://www.cloudera.com/documentation/enterprise/latest/topics/how_to_configure_cm_tls.html#concept...

[#3]

I made updates, but now I'm getting the following error message

[13/Oct/2018 00:09:48 +0000] 4701 MainThread agent        ERROR    Heartbeating to cloudera6.localdomain:7182 failed.
Traceback (most recent call last):
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1371, in _send_heartbeat
    response = self.requestor.request('heartbeat', heartbeat_data)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 141, in request
    return self.issue_request(call_request, message_name, request_datum)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 254, in issue_request
    call_response = self.transceiver.transceive(call_request)
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 483, in transceive
    result = self.read_framed_message()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 489, in read_framed_message
    framed_message = response_reader.read_framed_message()
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 417, in read_framed_message
    raise ConnectionClosedException("Reader read 0 bytes.")
ConnectionClosedException: Reader read 0 bytes.

I should add, I also added a line to the hosts file so that there is a non-Local IP address.

Before I followed the instructions to update the TLS values, I was getting this error message.

[#4]

@Meister1867 I am also getting the same error. Have you fixed this error? if yes, then please share the steps.

Thanks in advance.

[#5]

Did you slove ths problem?
I'm getting the same error now...
Have any suggestions?
Thanks a lot!

[#6]

I am also getting the same error. Have you fixed this error? if yes, then please share the steps.

Thanks in advance.

[#7]

@datasir,

We need to know what you have configured in Cloudera Manager with regard to agent communication (primarily agent encryption and authorization).

Also, check your Cloudera Manager log for messages at the same time as your agent error messages.

It would be a good idea to share the errors you are seeing so we can be sure we know what issue you are seeing.

Also, post your configuration for the agent having the problem.  This can be obtained with:

grep -v -e '^[[:space:]]*$' -e '^#' /etc/cloudera-scm-agent/config.ini

[#8]

I'm having the same issue. This is my agent configuration:

[General]
server_host=cloudera-1
server_port=7182
max_collection_wait_seconds=10.0
metrics_url_timeout_seconds=30.0
task_metrics_timeout_seconds=5.0
monitored_nodev_filesystem_types=nfs,nfs4,tmpfs
local_filesystem_whitelist=ext2,ext3,ext4,xfs
impala_profile_bundle_max_bytes=1073741824
stacks_log_bundle_max_bytes=1073741824
stacks_log_max_uncompressed_file_size_bytes=5242880
orphan_process_dir_staleness_threshold=5184000
orphan_process_dir_refresh_interval=3600
scm_debug=INFO
dns_resolution_collection_interval_seconds=60
dns_resolution_collection_timeout_seconds=30

This is my Cloudera Server config:

Link to the image:

https://imgur.com/SIciDFr

Regards,

Silva

[#9]

@JoaquinS Your CM agent config file does not have TLS enabled, but the CM server configuration has. 

I would suggest to disable the Use TLS Encryption for Agents, Use TLS Authentication of Agents to Server, Verify Agent Hostname Against Certificate settings and restart CM to proceed to make the installation work in a first step.

Once that is done, please follow the steps in this documentation chapter to enable TLS encryption for CM agent communication.