Hi,
Need help with understanding and resolving this Auto-TLS issue.
I have a CDH cluster (Version: Cloudera Enterprise 6.3.1) which runs fine without TLS.
But when I enable Auto-TLS, services do not start. I've followed the documentation about Auto-TLS and restarted services in the correct order, but still the services complain about JKS files being missing in the process directory.
Certmanager is created properly by Auto-TLS and I see all host, global certs, JKS, truststore etc.
But when I start any service, the process directory (which is dynamically created), these JKS files are not created there. I'm unable to figure-out which configuration could be causing this. My cloudera-scm-server / cloudera-scm-agent are healty after Auto-TLS restart.
Thanks in advance for any pointers.
---Details
cloudera-scm-agent is using JKS form this directly as per documentation.
/var/lib/cloudera-scm-agent/agent-cert
[root@cdh63 agent-cert]# ls -l
total 44
-rw-r--r-- 1 cloudera-scm cloudera-scm 1606 Jul 23 05:24 cm-auto-global_cacerts.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 1211 Jul 23 05:24 cm-auto-global_truststore.jks
-rw-r----- 1 cloudera-scm cloudera-scm 3277 Jul 23 05:24 cm-auto-host_cert_chain.pem
-rw------- 1 cloudera-scm cloudera-scm 5823 Jul 23 05:24 cm-auto-host_key_cert_chain.pem
-rw------- 1 cloudera-scm cloudera-scm 2546 Jul 23 05:24 cm-auto-host_key.pem
-rw------- 1 cloudera-scm cloudera-scm 43 Jul 23 05:24 cm-auto-host_key.pw
-rw------- 1 cloudera-scm cloudera-scm 4288 Jul 23 05:24 cm-auto-host_keystore.jks
-rw-r--r-- 1 cloudera-scm cloudera-scm 1606 Jul 23 05:24 cm-auto-in_cluster_ca_cert.pem
-rw-r--r-- 1 cloudera-scm cloudera-scm 1211 Jul 23 05:24 cm-auto-in_cluster_truststore.jks
--
Directory: /var/lib/cloudera-scm-server/certmanager
[root@cdh63 certmanager]# ls -l
total 8
drwx------ 4 cloudera-scm cloudera-scm 80 Jul 23 03:55 CMCA
-rw-r----- 1 cloudera-scm cloudera-scm 65 Jul 23 03:55 frozen_config.ini
-rwxr-xr-x 1 cloudera-scm cloudera-scm 144 Jul 23 03:55 generate_host_cert
drwx------ 4 cloudera-scm cloudera-scm 85 Jul 23 03:55 hosts-key-store
drwx------ 2 cloudera-scm cloudera-scm 140 Jul 23 03:55 private
drwxr-xr-x 2 cloudera-scm cloudera-scm 156 Jul 23 03:55 trust-store
trust-store/cm-auto-in_cluster_ca_cert.pem
trust-store/cm-auto-in_cluster_truststore.jks
trust-store/cm-auto-global_truststore.jks
trust-store/cm-auto-global_cacerts.pem
# find hosts-key-store/
hosts-key-store/
hosts-key-store/cdh63.myhostname.net
hosts-key-store/cdh63.myhostname.net/cm-auto-host_key.pem
hosts-key-store/cdh63.myhostname.net/cm-auto-host_cert_chain.pem
hosts-key-store/cdh63.myhostname.net/cm-auto-host_key_cert_chain.pem
hosts-key-store/cdh63.myhostname.net/cm-auto-host_key.pw
hosts-key-store/cdh63.myhostname.net/cm-auto-host_keystore.jks
Example: Zookeepr Start issue.
/run/cloudera-scm-agent/process/94-zookeeper-server/logs/stderr.log
+ export 'ZOOKEEPER_SERVER_OPTS=-Djava.net.preferIPv4Stack=true -Dzookeeper.log.file=zookeeper-cmf-zookeeper-SERVER-cdh63.myhostname.net.log -Dzook
eeper.log.dir=/var/log/zookeeper -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.rmi.port=9010 -Dcom.sun.management.jmxre
mote.authenticate=false -Dcom.sun.management.jmxremote.ssl=true -Dcom.sun.management.jmxremote.registry.ssl=true -Dcom.sun.management.jmxremote.s
sl.need.client.auth=true -Dcom.sun.management.jmxremote.ssl.config.file=jmxremote.properties.key -Djute.maxbuffer=4194304 -Dzookeeper.datadir.aut
ocreate=false -Xms1050673152 -Xmx1050673152 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/zookeeper_zookeeper-SERVER-492ffe2ba85a5d85635a
5d7079f503d7_pid2979.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh'
+ ZOOKEEPER_SERVER_OPTS='-Djava.net.preferIPv4Stack=true -Dzookeeper.log.file=zookeeper-cmf-zookeeper-SERVER-cdh63.myhostname.net.log -Dzookeeper.l
og.dir=/var/log/zookeeper -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.rmi.port=9010 -Dcom.sun.management.jmxremote.au
thenticate=false -Dcom.sun.management.jmxremote.ssl=true -Dcom.sun.management.jmxremote.registry.ssl=true -Dcom.sun.management.jmxremote.ssl.need
.client.auth=true -Dcom.sun.management.jmxremote.ssl.config.file=jmxremote.properties.key -Djute.maxbuffer=4194304 -Dzookeeper.datadir.autocreate
=false -Xms1050673152 -Xmx1050673152 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/zookeeper_zookeeper-SERVER-492ffe2ba85a5d85635a5d7079f
503d7_pid2979.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh'
+ exec /usr/java/jdk1.8.0_181-cloudera/bin/java -cp '/var/run/cloudera-scm-agent/process/94-zookeeper-server:/opt/cloudera/parcels/CDH-6.3.2-1.cd
h6.3.2.p0.1605554/lib/zookeeper/lib/log4j.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/zookeeper/build/*:/opt/cloudera/parcels/C
DH-6.3.2-1.cdh6.3.2.p0.1605554/lib/zookeeper/build/lib/*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/zookeeper/*:/opt/cloudera/parc
els/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/zookeeper/lib/*:/opt/cloudera/cm/lib/plugins/event-publish-6.3.1-shaded.jar:/opt/cloudera/cm/lib/plugins/
tt-instrumentation-6.3.1.jar' -Djava.net.preferIPv4Stack=true -Dzookeeper.log.file=zookeeper-cmf-zookeeper-SERVER-cdh63.myhostname.net.log -Dzookee
per.log.dir=/var/log/zookeeper -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.rmi.port=9010 -Dcom.sun.management.jmxremo
te.authenticate=false -Dcom.sun.management.jmxremote.ssl=true -Dcom.sun.management.jmxremote.registry.ssl=true -Dcom.sun.management.jmxremote.ssl
.need.client.auth=true -Dcom.sun.management.jmxremote.ssl.config.file=jmxremote.properties.key -Djute.maxbuffer=4194304 -Dzookeeper.datadir.autoc
reate=false -Xms1050673152 -Xmx1050673152 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/zookeeper_zookeeper-SERVER-492ffe2ba85a5d85635a5d
7079f503d7_pid2979.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh org.apache.zookeeper.server.quorum.QuorumPeer
Main /var/run/cloudera-scm-agent/process/94-zookeeper-server/zoo.cfg
Error: Exception thrown by the agent : java.io.FileNotFoundException: /var/run/cloudera-scm-agent/process/94-zookeeper-server/cm-auto-host_keysto
re.jks (No such file or directory)
sun.management.AgentConfigurationError: java.io.FileNotFoundException: /var/run/cloudera-scm-agent/process/94-zookeeper-server/cm-auto-host_keyst
ore.jks (No such file or directory)
at sun.management.jmxremote.ConnectorBootstrap.createSslRMIServerSocketFactory(ConnectorBootstrap.java:712)
at sun.management.jmxremote.ConnectorBootstrap.exportMBeanServer(ConnectorBootstrap.java:774)
at sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:468)
at sun.management.Agent.startAgent(Agent.java:262)
at sun.management.Agent.startAgent(Agent.java:452)
Caused by: java.io.FileNotFoundException: /var/run/cloudera-scm-agent/process/94-zookeeper-server/cm-auto-host_keystore.jks (No such file or dire
ctory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileInputStream.<init>(FileInputStream.java:93)
at sun.management.jmxremote.ConnectorBootstrap.createSslRMIServerSocketFactory(ConnectorBootstrap.java:684)
... 4 more
[26/Jul/2020 08:31:20 +0000] 3088 MainThread redactor INFO Started launcher: /opt/cloudera/cm-agent/service/zookeeper/zkserver.sh 1 /var/
lib/zookeeper
[26/Jul/2020 08:31:20 +0000] 3088 MainThread redactor INFO Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 3097
[26/Jul/2020 08:31:20 +0000] 3098 MainThread redactor INFO Re-exec redactor: /opt/cloudera/cm-agent/bin/cm redactor --fds 3 5
[26/Jul/2020 08:31:20 +0000] 3098 MainThread redactor INFO Started redactor
Sun Jul 26 08:31:20 UTC 2020
+ source_parcel_environment
+ '[' '!' -z /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/meta/cdh_env.sh ']'
