Hi,   Need help with understanding and resolving this Auto-TLS issue. I have a CDH cluster (Version: Cloudera Enterprise 6.3.1) which runs fine without TLS. But when I enable Auto-TLS, services do not start. I've followed the documentation about Auto-TLS and restarted services in the correct order, but still the services complain about JKS files being missing in the process directory.   Certmanager is created properly by Auto-TLS and I see all host, global certs, JKS, truststore etc. But when I start any service, the process directory (which is dynamically created), these JKS files are not created there. I'm unable to figure-out which configuration could be causing this. My cloudera-scm-server  / cloudera-scm-agent are healty after Auto-TLS restart.   Thanks in advance for any pointers.   ---Details cloudera-scm-agent is using JKS form this directly as per documentation.   /var/lib/cloudera-scm-agent/agent-cert [root@cdh63 agent-cert]# ls -l total 44 -rw-r--r-- 1 cloudera-scm cloudera-scm 1606 Jul 23 05:24 cm-auto-global_cacerts.pem -rw-r--r-- 1 cloudera-scm cloudera-scm 1211 Jul 23 05:24 cm-auto-global_truststore.jks -rw-r----- 1 cloudera-scm cloudera-scm 3277 Jul 23 05:24 cm-auto-host_cert_chain.pem -rw------- 1 cloudera-scm cloudera-scm 5823 Jul 23 05:24 cm-auto-host_key_cert_chain.pem -rw------- 1 cloudera-scm cloudera-scm 2546 Jul 23 05:24 cm-auto-host_key.pem -rw------- 1 cloudera-scm cloudera-scm 43 Jul 23 05:24 cm-auto-host_key.pw -rw------- 1 cloudera-scm cloudera-scm 4288 Jul 23 05:24 cm-auto-host_keystore.jks -rw-r--r-- 1 cloudera-scm cloudera-scm 1606 Jul 23 05:24 cm-auto-in_cluster_ca_cert.pem -rw-r--r-- 1 cloudera-scm cloudera-scm 1211 Jul 23 05:24 cm-auto-in_cluster_truststore.jks   -- Directory: /var/lib/cloudera-scm-server/certmanager [root@cdh63 certmanager]# ls -l total 8 drwx------ 4 cloudera-scm cloudera-scm 80 Jul 23 03:55 CMCA -rw-r----- 1 cloudera-scm cloudera-scm 65 Jul 23 03:55 frozen_config.ini -rwxr-xr-x 1 cloudera-scm cloudera-scm 144 Jul 23 03:55 generate_host_cert drwx------ 4 cloudera-scm cloudera-scm 85 Jul 23 03:55 hosts-key-store drwx------ 2 cloudera-scm cloudera-scm 140 Jul 23 03:55 private drwxr-xr-x 2 cloudera-scm cloudera-scm 156 Jul 23 03:55 trust-store   trust-store/cm-auto-in_cluster_ca_cert.pem trust-store/cm-auto-in_cluster_truststore.jks trust-store/cm-auto-global_truststore.jks trust-store/cm-auto-global_cacerts.pem   # find hosts-key-store/ hosts-key-store/ hosts-key-store/cdh63.myhostname.net hosts-key-store/cdh63.myhostname.net/cm-auto-host_key.pem hosts-key-store/cdh63.myhostname.net/cm-auto-host_cert_chain.pem hosts-key-store/cdh63.myhostname.net/cm-auto-host_key_cert_chain.pem hosts-key-store/cdh63.myhostname.net/cm-auto-host_key.pw hosts-key-store/cdh63.myhostname.net/cm-auto-host_keystore.jks     Example: Zookeepr Start issue.     /run/cloudera-scm-agent/process/94-zookeeper-server/logs/stderr.log   + export 'ZOOKEEPER_SERVER_OPTS=-Djava.net.preferIPv4Stack=true -Dzookeeper.log.file=zookeeper-cmf-zookeeper-SERVER-cdh63.myhostname.net.log -Dzook eeper.log.dir=/var/log/zookeeper -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.rmi.port=9010 -Dcom.sun.management.jmxre mote.authenticate=false -Dcom.sun.management.jmxremote.ssl=true -Dcom.sun.management.jmxremote.registry.ssl=true -Dcom.sun.management.jmxremote.s sl.need.client.auth=true -Dcom.sun.management.jmxremote.ssl.config.file=jmxremote.properties.key -Djute.maxbuffer=4194304 -Dzookeeper.datadir.aut ocreate=false -Xms1050673152 -Xmx1050673152 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/zookeeper_zookeeper-SERVER-492ffe2ba85a5d85635a 5d7079f503d7_pid2979.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh' + ZOOKEEPER_SERVER_OPTS='-Djava.net.preferIPv4Stack=true -Dzookeeper.log.file=zookeeper-cmf-zookeeper-SERVER-cdh63.myhostname.net.log -Dzookeeper.l og.dir=/var/log/zookeeper -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.rmi.port=9010 -Dcom.sun.management.jmxremote.au thenticate=false -Dcom.sun.management.jmxremote.ssl=true -Dcom.sun.management.jmxremote.registry.ssl=true -Dcom.sun.management.jmxremote.ssl.need .client.auth=true -Dcom.sun.management.jmxremote.ssl.config.file=jmxremote.properties.key -Djute.maxbuffer=4194304 -Dzookeeper.datadir.autocreate =false -Xms1050673152 -Xmx1050673152 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/zookeeper_zookeeper-SERVER-492ffe2ba85a5d85635a5d7079f 503d7_pid2979.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh' + exec /usr/java/jdk1.8.0_181-cloudera/bin/java -cp '/var/run/cloudera-scm-agent/process/94-zookeeper-server:/opt/cloudera/parcels/CDH-6.3.2-1.cd h6.3.2.p0.1605554/lib/zookeeper/lib/log4j.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/zookeeper/build/*:/opt/cloudera/parcels/C DH-6.3.2-1.cdh6.3.2.p0.1605554/lib/zookeeper/build/lib/*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/zookeeper/*:/opt/cloudera/parc els/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/zookeeper/lib/*:/opt/cloudera/cm/lib/plugins/event-publish-6.3.1-shaded.jar:/opt/cloudera/cm/lib/plugins/ tt-instrumentation-6.3.1.jar' -Djava.net.preferIPv4Stack=true -Dzookeeper.log.file=zookeeper-cmf-zookeeper-SERVER-cdh63.myhostname.net.log -Dzookee per.log.dir=/var/log/zookeeper -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.rmi.port=9010 -Dcom.sun.management.jmxremo te.authenticate=false -Dcom.sun.management.jmxremote.ssl=true -Dcom.sun.management.jmxremote.registry.ssl=true -Dcom.sun.management.jmxremote.ssl .need.client.auth=true -Dcom.sun.management.jmxremote.ssl.config.file=jmxremote.properties.key -Djute.maxbuffer=4194304 -Dzookeeper.datadir.autoc reate=false -Xms1050673152 -Xmx1050673152 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/zookeeper_zookeeper-SERVER-492ffe2ba85a5d85635a5d 7079f503d7_pid2979.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh org.apache.zookeeper.server.quorum.QuorumPeer Main /var/run/cloudera-scm-agent/process/94-zookeeper-server/zoo.cfg Error: Exception thrown by the agent : java.io.FileNotFoundException: /var/run/cloudera-scm-agent/process/94-zookeeper-server/cm-auto-host_keysto re.jks (No such file or directory) sun.management.AgentConfigurationError: java.io.FileNotFoundException: /var/run/cloudera-scm-agent/process/94-zookeeper-server/cm-auto-host_keyst ore.jks (No such file or directory) at sun.management.jmxremote.ConnectorBootstrap.createSslRMIServerSocketFactory(ConnectorBootstrap.java:712) at sun.management.jmxremote.ConnectorBootstrap.exportMBeanServer(ConnectorBootstrap.java:774) at sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:468) at sun.management.Agent.startAgent(Agent.java:262) at sun.management.Agent.startAgent(Agent.java:452) Caused by: java.io.FileNotFoundException: /var/run/cloudera-scm-agent/process/94-zookeeper-server/cm-auto-host_keystore.jks (No such file or dire ctory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.<init>(FileInputStream.java:138) at java.io.FileInputStream.<init>(FileInputStream.java:93) at sun.management.jmxremote.ConnectorBootstrap.createSslRMIServerSocketFactory(ConnectorBootstrap.java:684) ... 4 more [26/Jul/2020 08:31:20 +0000] 3088 MainThread redactor INFO Started launcher: /opt/cloudera/cm-agent/service/zookeeper/zkserver.sh 1 /var/ lib/zookeeper [26/Jul/2020 08:31:20 +0000] 3088 MainThread redactor INFO Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 3097 [26/Jul/2020 08:31:20 +0000] 3098 MainThread redactor INFO Re-exec redactor: /opt/cloudera/cm-agent/bin/cm redactor --fds 3 5 [26/Jul/2020 08:31:20 +0000] 3098 MainThread redactor INFO Started redactor Sun Jul 26 08:31:20 UTC 2020 + source_parcel_environment + '[' '!' -z /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/meta/cdh_env.sh ']'     ... View more