Hi Jerry,   Here is what I have on my system:   /etc/hadoop/conf.cloudera.yarn -r-------- 1 root hadoop  156 Jan 24 01:00 container-executor.cfg -rw-r--r-- 1 root root   3894 Jan 17 22:56 core-site.xml -rw-r--r-- 1 root root    617 Jan 17 22:56 hadoop-env.sh -rw-r--r-- 1 root root   2729 Jan 17 22:56 hdfs-site.xml   Even if I change above file permission, after start, it changes back to the same permission.   From manager I have this Container Executor Group = yarn   Upgrade also not allowing as it requires all services should be up and running.   Let me know if you need any more details.   Thanks, Tulasi   ... View more

Thanks Ben, will create a new thread. ... View more

I have followed the steps under "Configuring TLS/SSL for HDFS, YARN and MapReduce"   Service did not start successfully; not all of the required roles started: only 0/1 roles started. Reasons : Service has only 0 NodeManager roles running instead of minimum required 1   YARN failing to start and I see below error in the log Can't open /run/cloudera-scm-agent/process/190-yarn-NODEMANAGER/container-executor.cfg: Permission denied   This is the permission: -rw-r----- 1 yarn hadoop   997 Jan 18 02:22 creds.localjceks -rw------- 1 yarn hadoop  1746 Jan 18 02:22 yarn.keytab -r-------- 1 root hadoop   156 Jan 18 02:22 container-executor.cfg -rw------- 1 root root    3688 Jan 18 02:22 supervisor.conf   But after giving permission, restart creates another foldr with the same permission, how to resolve this problem.   Thanks, Tulasi   ... View more

@bgooley,  space characters at the bginning of cert/key in the agent configuration file is created this problem. After removing all of those spaces, restarted agent worked.   I didn't expect a space can create this sort of problem without telling what is going wrong.   Thanks for helping to figure this silly problem.     ... View more

Here is the output:   [root@optim-rhel72-uppu ~]# egrep -v '^[[:blank:]]*#|^$' /etc/cloudera-scm-agent/config.ini [General] server_host=optim-rhel72-uppu.development.unicomglobal.software server_port=7182 max_collection_wait_seconds=10.0 metrics_url_timeout_seconds=30.0 task_metrics_timeout_seconds=5.0 monitored_nodev_filesystem_types=nfs,nfs4,tmpfs local_filesystem_whitelist=ext2,ext3,ext4,xfs impala_profile_bundle_max_bytes=1073741824 stacks_log_bundle_max_bytes=1073741824 stacks_log_max_uncompressed_file_size_bytes=5242880 orphan_process_dir_staleness_threshold=5184000 orphan_process_dir_refresh_interval=3600 scm_debug=INFO dns_resolution_collection_interval_seconds=60 dns_resolution_collection_timeout_seconds=30 [Security] use_tls=1 max_cert_depth=9  verify_cert_file=/opt/cloudera/security/pki/optim-rhel72-uppu.pem  verify_cert_dir=/opt/cloudera/security/pki  client_key_file=/opt/cloudera/security/pki/agent.key  client_keypw_file=/etc/cloudera-scm-agent/agentkey.pw  client_cert_file=/opt/cloudera/security/pki/agent.pem [Hadoop] [Cloudera] [JDBC] [root@optim-rhel72-uppu ~]#   ... View more

Hi,   Here is my response to your questions, can you please correct me what I am doing wrong. Also if you need some more details, I should be able to share.   Thanks, Tulasi   1.) Ensure that the certificates are in a standard x509 format for the agent. Yes it is standard x509, see my response to Bgooley 2.) Ensure that the truststores/keystores on the CM host are in JCEKS format and not pkcs12. As per cloudera document, it should be JCEKS. From the link   https://www.cloudera.com/documentation/enterprise/5-15-x/topics/how_to_configure_cm_tls.html section "Generate TLS Certificate", point 3 3.) Make sure that the cloudera-scm user can read the Private Key, Certificates, Truststores, and Password Files. Yes, see my response to gzigldrum 4.) Make sure that the certificate on the failing agent contains a proper CN and DNS Alt Name if Alt Names are in use. Yes, I have verified this as well 5.) Are you using self-signed certificates or certificates signed by a CA? I am using self signed certificate 6.) If all else fails you can obtain a tcpdump of attempted communication with the server. The port that we normally heartbeat to is 7182. You can then review the conversation between the server and agent to attempt to identify at what point the error is returned and potentially what error is being observed at the protocol level. You can identify and restrict your tcpdump information by tcp.stream. [root@optim-rhel72-uppu ~]# tcpdump -i any 'port 7182' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes 21:20:03.562131 IP optim-rhel72-uppu.development.unicomglobal.software.44942 > optim-rhel72-uppu.development.unicomglobal.software.7182: Flags [S], seq 3560294529, win 43690, options [mss 65495,sackOK,TS val 1632415805 ecr 0,nop,wscale 7], length 0 21:20:03.562225 IP optim-rhel72-uppu.development.unicomglobal.software.44942 > optim-rhel72-uppu.development.unicomglobal.software.7182: Flags [.], ack 1, win 342, options [nop,nop,TS val 1632415805 ecr 1632415805], length 0 21:20:03.562549 IP optim-rhel72-uppu.development.unicomglobal.software.44942 > optim-rhel72-uppu.development.unicomglobal.software.7182: Flags [P.], seq 1:254, ack 1, win 342, options [nop,nop,TS val 1632415806 ecr 1632415805], length 253 21:20:03.587871 IP optim-rhel72-uppu.development.unicomglobal.software.44942 > optim-rhel72-uppu.development.unicomglobal.software.7182: Flags [.], ack 16390, win 1365, options [nop,nop,TS val 1632415831 ecr 1632415831], length 0 21:20:03.587919 IP optim-rhel72-uppu.development.unicomglobal.software.44942 > optim-rhel72-uppu.development.unicomglobal.software.7182: Flags [.], ack 21184, win 2388, options [nop,nop,TS val 1632415831 ecr 1632415831], length 0 21:20:03.619895 IP optim-rhel72-uppu.development.unicomglobal.software.44942 > optim-rhel72-uppu.development.unicomglobal.software.7182: Flags [P.], seq 254:516, ack 21184, win 2388, options [nop,nop,TS val 1632415863 ecr 1632415831], length 262 21:20:03.628945 IP optim-rhel72-uppu.development.unicomglobal.software.44942 > optim-rhel72-uppu.development.unicomglobal.software.7182: Flags [F.], seq 516, ack 21185, win 2388, options [nop,nop,TS val 1632415872 ecr 1632415864], length 0 ... View more

Hi, Thanks for the response, I see one difference (in point 2) between document what you have noted. Document says to use PKCS12 format but you are suggesting JCEKS format. I didn't try with JCEKS, could you please confirm which format is correct. 2.) Ensure that the truststores/keystores on the CM host are in JCEKS format and not pkcs12. Document link https://www.cloudera.com/documentation/enterprise/5-15-x/topics/how_to_configure_cm_tls.html Thanks, Tulasi ... View more

It is still not solved, any inputs would be appreciated.   Thanks, Tulasi ... View more

Thanks for looking at it, here is the output and let me know if anything else is required.   [root@optim-rhel72-uppu ~]# namei -l /opt/cloudera/security/pki/agent.key f: /opt/cloudera/security/pki/agent.key dr-xr-xr-x root         root         / drwxr-xr-x root         root         opt drwxr-xr-x cloudera-scm cloudera-scm cloudera drwxr-xr-x cloudera-scm cloudera-scm security drwxr-xr-x cloudera-scm cloudera-scm pki lrwxrwxrwx cloudera-scm cloudera-scm agent.key -> optim-rhel72-uppu.key -rw-r--r-- cloudera-scm cloudera-scm   optim-rhel72-uppu.key [root@optim-rhel72-uppu ~]# namei -l /etc/cloudera-scm-agent/agentkey.pw f: /etc/cloudera-scm-agent/agentkey.pw dr-xr-xr-x root root / drwxr-xr-x root root etc drwxr-xr-x root root cloudera-scm-agent -rw-r--r-- root root agentkey.pw ... View more