Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

issue configuring TLS level 3

Highlighted

issue configuring TLS level 3

Contributor

helo I was following the guide in here http://www.cloudera.com/documentation/enterprise/5-5-x/topics/cm_sg_config_tls_agent_auth.html#conce...

 

I added the verify_cert_dir the client_key_file the client_keypwd_file and the client_cert_files but, I'm getting some errors after I restart :

 

ERROR heartbeating to rofl.dude.c1:7182 failed/

Trackeback (most recen call last):

file "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.7.0-py2.7.egg/cmf/agent.py", line 1188, in _send_heartbeat

SSLError: unexpected eof

 

I'm using centos 7 Cloudera Manager 5.7.0

8 REPLIES 8

Re: issue configuring TLS level 3

Super Guru

Hello,

 

First, is that the entire call stack for the exception or did you truncate it?  It seems a tad small.

 

Since you are using a "verify_cert_dir" I would check to make sure your hash links are set up properly. 

Check the documentation here for specifics:

 

http://www.cloudera.com/documentation/enterprise/5-7-x/topics/cm_sg_config_tls_auth.html#topic_3_3

 

If it looks correct and you still have the problem, please give us an "ls -l" in the "verify_cert_dir" directory so we can double check.

 

Regards,

 

Ben

Re: issue configuring TLS level 3

New Contributor

Im facing same issue please help me out, below is the error... 

 

File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.9.0-py2.6.egg/cmf/agent.py", line 1346, in _send_heartbeat
self.max_cert_depth)
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.9.0-py2.6.egg/cmf/https.py", line 132, in __init__
self.conn.connect()
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/M2Crypto-0.21.1-py2.6-linux-x86_64.egg/M2Crypto/httpslib.py", line 50, in connect
self.sock.connect((self.host, self.port))
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/M2Crypto-0.21.1-py2.6-linux-x86_64.egg/M2Crypto/SSL/Connection.py", line 185, in connect
ret = self.connect_ssl()
File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/M2Crypto-0.21.1-py2.6-linux-x86_64.egg/M2Crypto/SSL/Connection.py", line 178, in connect_ssl
return m2.ssl_connect(self.ssl)
SSLError: unexpected eof

Re: issue configuring TLS level 3

Expert Contributor

Hello,

 

If you are attempting to use verify_cert_dir with a non-single root certificate please make sure that you have followed our instructions to create the certificate hashes. If the hashes have not been created openssl will have difficulty performing certificate validation.

 

https://www.cloudera.com/documentation/enterprise/5-9-x/topics/cm_sg_config_tls_auth.html#topic_3_3_...

 

In most cases Unexpected end of file in regards to TLS means that a certificate was not available for transmission. For example:

 

The certificate has the wrong permissions and can't be read.

The certificate has the wrong permissions and can't be written.

The folder has the wrong permissions.

The certificate is improperly encoded.

The certificate is encrypted.

Customer Operations Engineer | Security SME | Cloudera, Inc.

Re: issue configuring TLS level 3

New Contributor

Hello  Ihebert,

 

Thanks a lot for the reply...actually we are using  "verify_cert_file=/opt/cloudera/security/pki/cdhmastertest.it.local-server.cert.pem" instead of verify_cert_dir  

 

 Please find the below permission of the file 

 

drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Jan 27 19:09 CAcerts
drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Feb 10 19:56 jks
drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Feb 3 16:03 pki
-rw-r--r-- 1 cloudera-scm cloudera-scm 1031 Jan 30 14:39 selfsigned.cer
drwxr-xr-x 2 cloudera-scm cloudera-scm 4096 Jan 27 19:09 x509

 

 

FIles inside jks folder 

 

[root@cdhmastertest jks]# ll
total 8
-rw-r--r-- 1 cloudera-scm cloudera-scm 2082 Jan 30 14:38 cdhmastertest1.keystore
-rw-r--r-- 1 cloudera-scm cloudera-scm 2246 Jan 27 15:26 cmhost-keystore.jks
[root@cdhmastertest jks]#

 

Now when I start service cloudera-scm-server start   I get  cloudera-scm-server dead but pid file exists

 

please find the below log of /var/log/cloudera-scm-server/cloudera-scm-server.log

 

 

 

 

2017-02-10 19:48:57,359 ERROR MainThread:com.cloudera.server.cmf.Main: Failed to start Agent listener.
2017-02-10 19:48:57,360 ERROR MainThread:com.cloudera.server.cmf.Main: Server failed.
org.apache.avro.AvroRuntimeException: java.io.IOException: Keystore was tampered with, or password was incorrect
at com.cloudera.server.common.HttpConnectorServer.start(HttpConnectorServer.java:89)
at com.cloudera.server.cmf.Main.startAgentServer(Main.java:572)
at com.cloudera.server.cmf.Main.startAvro(Main.java:483)
at com.cloudera.server.cmf.Main.run(Main.java:620)
at com.cloudera.server.cmf.Main.main(Main.java:217)
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772)
at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:55)
at java.security.KeyStore.load(KeyStore.java:1214)
at org.mortbay.jetty.security.SslSelectChannelConnector.createSSLContext(SslSelectChannelConnector.java:664)
at org.mortbay.jetty.security.SslSelectChannelConnector.doStart(SslSelectChannelConnector.java:613)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.Server.doStart(Server.java:235)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at com.cloudera.server.common.HttpConnectorServer.start(HttpConnectorServer.java:87)
... 4 more
Caused by: java.security.UnrecoverableKeyException: Password verification failed
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:770)
... 12 more
2017-02-10 19:48:57,364 INFO Thread-11:org.springframework.context.support.ClassPathXmlApplicationContext: Closing ApplicationContext 'rootContext': startup date [Fri Feb 10 19:48:09 IST 2017]; parent: org.springframework.context.support.GenericApplicationContext@5f91e550

 

 Any help would be really appriciated ... thanks in advance

Re: issue configuring TLS level 3

Expert Contributor

Hello,

 

Thank you for provide error log data, the information you have provided suggest that there is a problem with the Keystore or truststore you have configured for us in Cloudera Manager.

 

org.apache.avro.AvroRuntimeException: java.io.IOException: Keystore was tampered with, or password was incorrect

 

The error shown here means that the password for the keystore or truststore is invalid and does not match what you have configured. Unfortunately recovering from this can be difficult as it requires modifications to configurations within the CM database. If you are licensed customer I recommend that you open a case with us.

Customer Operations Engineer | Security SME | Cloudera, Inc.

Re: issue configuring TLS level 3

New Contributor

Hello Ihebert,

 

Thanks alot for the reply , unfortunately we are not licensed customer :(  Please help me someway if possible

I just logged into postgresql, not sure which database I shuould check for changing the keystore password like you suggested. Below are the databases

 

public | audits | table | scm
public | client_configs | table | scm
public | client_configs_to_hosts | table | scm
public | cluster_activated_releases | table | scm
public | cluster_activated_releases_aud | table | scm
public | cluster_managed_releases | table | scm
public | cluster_undistributed_releases | table | scm
public | clusters | table | scm
public | clusters_aud | table | scm
public | cm_peers | table | scm
public | cm_version | table | scm
public | command_schedules | table | scm
public | commands | table | scm
public | config_containers | table | scm
public | configs | table | scm
public | configs_aud | table | scm
public | credentials | table | scm
public | diagnostics_events | table | scm
public | external_accounts | table | scm
public | external_accounts_aud | table | scm
public | global_settings | table | scm
public | host_template_to_role_conf_grp | table | scm
public | host_templates | table | scm
public | hosts | table | scm
public | hosts_aud | table | scm
public | metrics | table | scm
public | parcel_components | table | scm
public | parcels | table | scm
public | process_active_releases | table | scm
public | processes | table | scm
public | releases | table | scm
public | releases_aud | table | scm
public | revisions | table | scm
public | role_config_groups | table | scm
public | role_config_groups_aud | table | scm
public | role_staleness_status | table | scm
public | roles | table | scm
public | roles_aud | table | scm
public | schema_version | table | scm
public | services | table | scm
public | services_aud | table | scm
public | snapshot_policies | table | scm
public | user_roles | table | scm
public | user_settings | table | scm
public | users | table | scm

 

Do we have any way to get back clusters pervious state so that we can do the TLS/SSL all over again :(

 

Re: issue configuring TLS level 3

Expert Contributor

Hello,

 

I have sent you a private message. We tend to avoid providing database manipulation instructions like this publicly to protect users. 

 

In releases above 5.4.3 you should be able to reach port 7180 and disable TLS or reconfigure things but occasionally the failures are fatal. Whe nthe failures are fatal they will prevent the server from starting successfully and you will need to manualy disable TLS on CM.

Customer Operations Engineer | Security SME | Cloudera, Inc.

Re: issue configuring TLS level 3

New Contributor

I am also facing this issue [SSLError: unexpected eof] while configuring tls level 3

 

https://community.cloudera.com/t5/Cloudera-Manager-Installation/SSLError-unexpected-eof-error-while-...