question Re: After enabling TLS cloudera agent heartbeat failing in Support Questions

After enabling TLS cloudera agent heartbeat failing

Tulasi — Fri, 28 Dec 2018 05:09:38 GMT

Version: Cloudera Express 5.15.0

Java VM Name: Java HotSpot(TM) 64-Bit Server VM

Java VM Vendor: Oracle Corporation

Java Version: 1.7.0_67

System details:

Linux optim-rhel72-uppu.development.unicomglobal.software 3.10.0-327.28.3.el7.x86_64 #1 SMP Fri Aug 12 13:21:05 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux

It is a single host and I am using self signed certificate. I am just validating a POC with one of my product and hence not yet licensed.

Followed the steps mentioned at this link:

https://www.cloudera.com/documentation/enterprise/5-11-x/topics/how_to_configure_cm_tls.html

https://www.cloudera.com/documentation/enterprise/5-15-x/topics/sg_self_signed_tls.html

After enabling TLS, cloudera agant heartbeat is failing with the below lines in the cloudera-scm-agent.log

[27/Dec/2018 20:58:28 +0000] 6869 MainThread agent        ERROR    Heartbeating to optim-rhel72-uppu.development.unicomglobal.software:7182 failed.
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.15.0-py2.7.egg/cmf/agent.py", line 1424, in _send_heartbeat
    self.max_cert_depth)
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.15.0-py2.7.egg/cmf/https.py", line 138, in __init__
    self.conn.connect()
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/M2Crypto-0.24.0-py2.7-linux-x86_64.egg/M2Crypto/httpslib.py", line 59, in connect
    sock.connect((self.host, self.port))
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/M2Crypto-0.24.0-py2.7-linux-x86_64.egg/M2Crypto/SSL/Connection.py", line 195, in connect
    ret = self.connect_ssl()
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/M2Crypto-0.24.0-py2.7-linux-x86_64.egg/M2Crypto/SSL/Connection.py", line 188, in connect_ssl
    return m2.ssl_connect(self.ssl, self._timeout)
SSLError: unexpected eof

Below lines in the cloudera-scm-server.log

2018-12-27 20:58:13,025 WARN 1320793343@agentServer-16:org.mortbay.log: javax.net.ssl.SSLHandshakeException: null cert chain
2018-12-27 20:58:28,034 WARN 1320793343@agentServer-16:org.mortbay.log: javax.net.ssl.SSLHandshakeException: null cert chain
2018-12-27 20:58:43,447 WARN 1320793343@agentServer-16:org.mortbay.log: javax.net.ssl.SSLHandshakeException: null cert chain
2018-12-27 20:58:58,082 WARN 1320793343@agentServer-16:org.mortbay.log: javax.net.ssl.SSLHandshakeException: null cert chain
2018-12-27 20:59:13,140 WARN 1320793343@agentServer-16:org.mortbay.log: javax.net.ssl.SSLHandshakeException: null cert chain

I have tried multiple times but none of them working.

I didn't find any error while running this command:

openssl s_client -showcerts -connect optim-rhel72-uppu.development.unicomglobal.software:7182

Any help would be highly appreciated.

Thanks,

Tulasi

Re: After enabling TLS cloudera agent heartbeat failing

bgooley — Fri, 28 Dec 2018 19:46:58 GMT

Hi @Tulasi,

The exception in the agent and error in CM indicate that the agent is not presenting its certificate for authentication properly. This is often due to something being wrong regarding your client_key_file configuration in /etc/cloudera-scm-agent/config.ini.

Can you check to see what is configured with and then verify the contents and that it is a valid PEM formatted key?

You can try this on the agent host to verify your agent has the right key and key password configured:

# openssl rsa -in `grep client_key_file /etc/cloudera-scm-agent/config.ini | sed 's/client_key_file=//'` -check -passin pass:`grep client_keypw_file /etc/cloudera-scm-agent/config.ini |sed 's/client_keypw_file=//'| xargs cat`

The output should show "RSA key ok" and the base64 encoded key text.

Re: After enabling TLS cloudera agent heartbeat failing

Tulasi — Sat, 29 Dec 2018 03:59:56 GMT

Hi Bgooley,

Appreciate your quick response.

I have tried what you have suggested and the output also matching, see below:

[root@optim-rhel72-uppu security]# openssl rsa -in `grep client_key_file /etc/cloudera-scm-agent/config.ini | sed 's/client_key_file=//'` -check -passin pass:`grep client_keypw_file /etc/cloudera-scm-agent/config.ini |sed 's/client_keypw_file=//'| xargs cat`
RSA key ok
writing RSA key
-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEA5BoT3b00kTdMvhv9UO2Na3//1n+HNcD9nxH4ZVW/Ye2HeY+3

<<<< other lines>>>>

C5bNRh3+6bdksCaDFXB85cm96iZfmbZIO4oks5fomkuvBpPa3izjAQ==
-----END RSA PRIVATE KEY-----

[root@optim-rhel72-uppu security]#

Let me know if you need anything else.

Thanks,

Tulasi

Re: After enabling TLS cloudera agent heartbeat failing

gzigldrum — Sat, 29 Dec 2018 13:58:57 GMT

This shows the client_key_file settings are fine, please also verify that the CM agent can access the files referenced as client_key_file and client_key_file in /etc/cloudera-scm-agent/config.ini, e.g. with command

namei -l <path-to-file-name>

Re: After enabling TLS cloudera agent heartbeat failing

Tulasi — Sat, 29 Dec 2018 14:10:08 GMT

This is what I see:

[root@optim-rhel72-uppu ~]# namei -l /opt/cloudera/security/pki
f: /opt/cloudera/security/pki
dr-xr-xr-x root root /
drwxr-xr-x root root opt
drwxr-xr-x cloudera-scm cloudera-scm cloudera
drwxr-xr-x cloudera-scm cloudera-scm security
drwxr-xr-x cloudera-scm cloudera-scm pki

[root@optim-rhel72-uppu ~]# namei -l /etc/cloudera-scm-agent
f: /etc/cloudera-scm-agent
dr-xr-xr-x root root /
drwxr-xr-x root root etc
drwxr-xr-x root root cloudera-scm-agent

Re: After enabling TLS cloudera agent heartbeat failing

gzigldrum — Sat, 29 Dec 2018 15:35:10 GMT

Sorry for being unclear but the command needs to have the full path including the actual filename as specified in config.ini for client_key_file and client_keypw_file

Re: After enabling TLS cloudera agent heartbeat failing

Tulasi — Sat, 29 Dec 2018 16:14:08 GMT

Thanks for looking at it, here is the output and let me know if anything else is required.

[root@optim-rhel72-uppu ~]# namei -l /opt/cloudera/security/pki/agent.key
f: /opt/cloudera/security/pki/agent.key
dr-xr-xr-x root         root         /
drwxr-xr-x root         root         opt
drwxr-xr-x cloudera-scm cloudera-scm cloudera
drwxr-xr-x cloudera-scm cloudera-scm security
drwxr-xr-x cloudera-scm cloudera-scm pki
lrwxrwxrwx cloudera-scm cloudera-scm agent.key -> optim-rhel72-uppu.key
-rw-r--r-- cloudera-scm cloudera-scm   optim-rhel72-uppu.key
[root@optim-rhel72-uppu ~]# namei -l /etc/cloudera-scm-agent/agentkey.pw
f: /etc/cloudera-scm-agent/agentkey.pw
dr-xr-xr-x root root /
drwxr-xr-x root root etc
drwxr-xr-x root root cloudera-scm-agent
-rw-r--r-- root root agentkey.pw

Re: After enabling TLS cloudera agent heartbeat failing

Tulasi — Fri, 04 Jan 2019 09:05:28 GMT

It is still not solved, any inputs would be appreciated.

Thanks,

Tulasi

Re: After enabling TLS cloudera agent heartbeat failing

lhebert — Tue, 08 Jan 2019 16:39:23 GMT

Hi,

We are sorry you are still encountering this issue. Since we know that the problem is isolated to TLS and that the agent is reporting a null certificate chain you will need to isolate why the certificate chain is null.

1.) Ensure that the certificates are in a standard x509 format for the agent.

2.) Ensure that the truststores/keystores on the CM host are in JCEKS format and not pkcs12.

3.) Make sure that the cloudera-scm user can read the Private Key, Certificates, Truststores, and Password Files.

4.) Make sure that the certificate on the failing agent contains a proper CN and DNS Alt Name if Alt Names are in use.

5.) Are you using self-signed certificates or certificates signed by a CA?

6.) If all else fails you can obtain a tcpdump of attempted communication with the server. The port that we normally heartbeat to is 7182. You can then review the conversation between the server and agent to attempt to identify at what point the error is returned and potentially what error is being observed at the protocol level. You can identify and restrict your tcpdump information by tcp.stream.

Re: After enabling TLS cloudera agent heartbeat failing

Tulasi — Mon, 14 Jan 2019 09:05:46 GMT

Hi, Thanks for the response, I see one difference (in point 2) between document what you have noted. Document says to use PKCS12 format but you are suggesting JCEKS format. I didn't try with JCEKS, could you please confirm which format is correct. 2.) Ensure that the truststores/keystores on the CM host are in JCEKS format and not pkcs12. Document link https://www.cloudera.com/documentation/enterprise/5-15-x/topics/how_to_configure_cm_tls.html Thanks, Tulasi

Re: After enabling TLS cloudera agent heartbeat failing

gzigldrum — Mon, 14 Jan 2019 09:19:27 GMT

Please note that keystores/truststores need to be in standard JCEKS format and the documentation does not state otherwise. PKCS12 is only used for exporting and converting certificate plus private key to PEM format for CM agent config. Can you please point to the sentence suggesting PKCS12 format so that we can correct it?

Re: After enabling TLS cloudera agent heartbeat failing

Tulasi — Tue, 15 Jan 2019 05:24:41 GMT

Hi,

Here is my response to your questions, can you please correct me what I am doing wrong. Also if you need some more details, I should be able to share.

Thanks,

Tulasi

1.) Ensure that the certificates are in a standard x509 format for the agent.
Yes it is standard x509, see my response to Bgooley

2.) Ensure that the truststores/keystores on the CM host are in JCEKS format and not pkcs12.
As per cloudera document, it should be JCEKS. From the link
https://www.cloudera.com/documentation/enterprise/5-15-x/topics/how_to_configure_cm_tls.html
section "Generate TLS Certificate", point 3

3.) Make sure that the cloudera-scm user can read the Private Key, Certificates, Truststores, and Password Files.
Yes, see my response to gzigldrum

4.) Make sure that the certificate on the failing agent contains a proper CN and DNS Alt Name if Alt Names are in use.
Yes, I have verified this as well

5.) Are you using self-signed certificates or certificates signed by a CA?
I am using self signed certificate

6.) If all else fails you can obtain a tcpdump of attempted communication with the server. The port that we normally heartbeat to is 7182. You can then review the conversation between the server and agent to attempt to identify at what point the error is returned and potentially what error is being observed at the protocol level. You can identify and restrict your tcpdump information by tcp.stream.

[root@optim-rhel72-uppu ~]# tcpdump -i any 'port 7182'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
21:20:03.562131 IP optim-rhel72-uppu.development.unicomglobal.software.44942 > optim-rhel72-uppu.development.unicomglobal.software.7182: Flags [S], seq 3560294529, win 43690, options [mss 65495,sackOK,TS val 1632415805 ecr 0,nop,wscale 7], length 0
21:20:03.562225 IP optim-rhel72-uppu.development.unicomglobal.software.44942 > optim-rhel72-uppu.development.unicomglobal.software.7182: Flags [.], ack 1, win 342, options [nop,nop,TS val 1632415805 ecr 1632415805], length 0
21:20:03.562549 IP optim-rhel72-uppu.development.unicomglobal.software.44942 > optim-rhel72-uppu.development.unicomglobal.software.7182: Flags [P.], seq 1:254, ack 1, win 342, options [nop,nop,TS val 1632415806 ecr 1632415805], length 253
21:20:03.587871 IP optim-rhel72-uppu.development.unicomglobal.software.44942 > optim-rhel72-uppu.development.unicomglobal.software.7182: Flags [.], ack 16390, win 1365, options [nop,nop,TS val 1632415831 ecr 1632415831], length 0
21:20:03.587919 IP optim-rhel72-uppu.development.unicomglobal.software.44942 > optim-rhel72-uppu.development.unicomglobal.software.7182: Flags [.], ack 21184, win 2388, options [nop,nop,TS val 1632415831 ecr 1632415831], length 0
21:20:03.619895 IP optim-rhel72-uppu.development.unicomglobal.software.44942 > optim-rhel72-uppu.development.unicomglobal.software.7182: Flags [P.], seq 254:516, ack 21184, win 2388, options [nop,nop,TS val 1632415863 ecr 1632415831], length 262
21:20:03.628945 IP optim-rhel72-uppu.development.unicomglobal.software.44942 > optim-rhel72-uppu.development.unicomglobal.software.7182: Flags [F.], seq 516, ack 21185, win 2388, options [nop,nop,TS val 1632415872 ecr 1632415864], length 0

Re: After enabling TLS cloudera agent heartbeat failing

gzigldrum — Wed, 16 Jan 2019 11:18:14 GMT

In addition to that, can you please show us the CM agent configuration with

# egrep -v '^[[:blank:]]*#|^$' /etc/cloudera-scm-agent/config.ini

Re: After enabling TLS cloudera agent heartbeat failing

Tulasi — Wed, 16 Jan 2019 16:33:59 GMT

Here is the output:

[root@optim-rhel72-uppu ~]# egrep -v '^[[:blank:]]*#|^$' /etc/cloudera-scm-agent/config.ini
[General]
server_host=optim-rhel72-uppu.development.unicomglobal.software
server_port=7182
max_collection_wait_seconds=10.0
metrics_url_timeout_seconds=30.0
task_metrics_timeout_seconds=5.0
monitored_nodev_filesystem_types=nfs,nfs4,tmpfs
local_filesystem_whitelist=ext2,ext3,ext4,xfs
impala_profile_bundle_max_bytes=1073741824
stacks_log_bundle_max_bytes=1073741824
stacks_log_max_uncompressed_file_size_bytes=5242880
orphan_process_dir_staleness_threshold=5184000
orphan_process_dir_refresh_interval=3600
scm_debug=INFO
dns_resolution_collection_interval_seconds=60
dns_resolution_collection_timeout_seconds=30
[Security]
use_tls=1
max_cert_depth=9
verify_cert_file=/opt/cloudera/security/pki/optim-rhel72-uppu.pem
verify_cert_dir=/opt/cloudera/security/pki
client_key_file=/opt/cloudera/security/pki/agent.key
client_keypw_file=/etc/cloudera-scm-agent/agentkey.pw
client_cert_file=/opt/cloudera/security/pki/agent.pem
[Hadoop]
[Cloudera]
[JDBC]
[root@optim-rhel72-uppu ~]#

Re: After enabling TLS cloudera agent heartbeat failing

bgooley — Wed, 16 Jan 2019 21:42:45 GMT

@Tulasi,

Thank you for providing your config. It appears you have space characters at the beginning of your cert/key configs. Remove the space characters form the beginning of the following lines and then restart the agent:

verify_cert_file=/opt/cloudera/security/pki/optim-rhel72-uppu.pem
verify_cert_dir=/opt/cloudera/security/pki
client_key_file=/opt/cloudera/security/pki/agent.key
client_keypw_file=/etc/cloudera-scm-agent/agentkey.pw
client_cert_file=/opt/cloudera/security/pki/agent.pem

Re: After enabling TLS cloudera agent heartbeat failing

bgooley — Wed, 16 Jan 2019 21:46:50 GMT

NOTE:

You should only specify verify_cert_dir OR verify_cert_file, not both

Since you have a pem file, I would suggest using verify_cert_file and commenting out "verify_cert_dir=/opt/cloudera/security/pki"

Re: After enabling TLS cloudera agent heartbeat failing

Tulasi — Fri, 18 Jan 2019 04:22:47 GMT

@bgooley,

space characters at the bginning of cert/key in the agent configuration file is created this problem. After removing all of those spaces, restarted agent worked.

I didn't expect a space can create this sort of problem without telling what is going wrong.

Thanks for helping to figure this silly problem.

Re: After enabling TLS cloudera agent heartbeat failing

Tulasi — Fri, 18 Jan 2019 10:25:49 GMT

I have followed the steps under "Configuring TLS/SSL for HDFS, YARN and MapReduce"

Service did not start successfully; not all of the required roles started: only 0/1 roles started. Reasons : Service has only 0 NodeManager roles running instead of minimum required 1

YARN failing to start and I see below error in the log

Can't open /run/cloudera-scm-agent/process/190-yarn-NODEMANAGER/container-executor.cfg: Permission denied

This is the permission:

-rw-r----- 1 yarn hadoop   997 Jan 18 02:22 creds.localjceks
-rw------- 1 yarn hadoop 1746 Jan 18 02:22 yarn.keytab
-r-------- 1 root hadoop   156 Jan 18 02:22 container-executor.cfg
-rw------- 1 root root    3688 Jan 18 02:22 supervisor.conf

But after giving permission, restart creates another foldr with the same permission, how to resolve this problem.

Thanks,

Tulasi

Re: After enabling TLS cloudera agent heartbeat failing

bgooley — Wed, 23 Jan 2019 21:47:18 GMT

I opened a Jira internally at Cloudera to ask that config.ini leading non-word characters be trimmed.

Regards,

Ben

Re: After enabling TLS cloudera agent heartbeat failing

bgooley — Wed, 23 Jan 2019 21:48:57 GMT

@Tulasi,

Could you start a new thread with your new issue so that we don't mix issues in the same thread. the space character issue is likely to help others, so it would be good to start a new thread for permission denied issue. I think it is a known one, but it will be easier to discuss if we can start fresh.