First, appreciate you looking into this.
Just left verify_cert_file in place in /etc/cloudera-scm-agent/config.ini and same issue occurs.
Quite certain I got the certs incorrectly configured. The reason why I'm thinking that is because I pointed agent.pem to various .pem files I have in the above-said folder. Same result.
I also have an HAproxy / Keepalived configuration pointing to a common VIP between the NN's called cm-c01.mws.mds.xyz . That works fine for the CM UI, however, I'm trying the agent connection to the primary server directly at first to get that working before trying via the VIP. So for the purpose of this exercise, I'm not using this VIP.
Here's my cert process:
1) Generate a cert.
keytool -genkeypair -alias $(hostname -f) -keyalg RSA -keysize 2048 -dname "cn=$(hostname -f),OU=MDS,O=MDS,L=Los Angeles,ST=California,C=US" -keypass $(hostname -f) -keystore $(hostname -f).keystore.jks -storepass $(hostname -f) -validity 3650 -ext EKU=serverAuth,clientAuth,codeSigning,emailProtection,timeStamping,OCSPSigning -ext san=dns:$(hostname -f)
2) Export cert from keystore.
keytool -export -alias $(hostname -f) -keystore $(hostname -f).keystore.jks -rfc -file $(hostname -f)-selfsigned.cer
3) Import the cert into the java truststores.
# Import root ca to the JDK Truststore
keytool -import -alias $(hostname -f) -file /opt/cloudera/security/jks/$(hostname -f)-selfsigned.cer -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit
# Import root ca to the JDK Truststore
keytool -import -alias $(hostname -f) -file /opt/cloudera/security/jks/$(hostname -f)-selfsigned.cer -keystore /usr/java/jdk1.8.0_181-cloudera/jre/lib/security/jssecacerts -storepass changeit
4) Copy the files around and verify.
cp -ip /opt/cloudera/security/x509/$(hostname -f).pem /opt/cloudera/security/pki/$(hostname -f).pem;
cp -ip /opt/cloudera/security/x509/$(hostname -f).pem /opt/cloudera/security/pki/;
cp -ip /opt/cloudera/security/jks/$(hostname -f).keystore.jks /opt/cloudera/security/pki;
ln -s /opt/cloudera/security/pki/$(hostname -f).keystore.jks /opt/cloudera/security/pki/server.jks
ln -s /opt/cloudera/security/pki/$(hostname -f).pem /opt/cloudera/security/pki/agent.pem
openssl x509 -in /opt/cloudera/security/pki/$(hostname -f).pem -noout -text
5) Repeat the above on the other hosts.
6) HTTPS / 7183 works fine from CM, just not the agent to server communication, now that I have it enabled.
The parts I'm not clear about stems from my intermediate knowledge of certs:
A) Do I copy the generated cert off the CM / NN server to the other servers in the cluster?
B) Should I be exporting the private key from the SSL keystore by I) converting to pcs12 II) exporting the cert and key from that? One of the dozens of options I tried included this step but that didn't work out either when I was trying the connectivity of the cloudera-scm-agent on the NN instead of any other participating host:
keytool -importkeystore \
-srckeystore cm-r01nn01.mws.mds.xyz.keystore.jks \
-destkeystore cm-r01nn01.mws.mds.xyz.keystore.p12 \
-deststoretype PKCS12 \
-srcalias cm-c01.mws.mds.xyz \
-deststorepass cm-r01nn01.mws.mds.xyz \
-destkeypass cm-r01nn01.mws.mds.xyz
# openssl pkcs12 -in cm-r01nn01.mws.mds.xyz.keystore.p12 -nokeys -out cm-r01nn01.mws.mds.xyz.cert.pem
Enter Import Password:
MAC verified OK
#
# openssl pkcs12 -in cm-r01nn01.mws.mds.xyz.keystore.p12 -nodes -nocerts -out cm-r01nn01.mws.mds.xyz.key.pem
Enter Import Password:
MAC verified OK
#Thinking I might have the commands right, just not the sequence or I'm missing a step or two. Further notes and info:
[ UTILITY NODE ]
[root@cm-r01en01 pki]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts|grep cm
Enter keystore password: changeit
acraizfnmt-rcm, Mar 26, 2019, trustedCertEntry,
cm-r01nn02.mws.mds.xyz, Apr 14, 2019, trustedCertEntry,
cm-r01en01.mws.mds.xyz, Apr 1, 2019, trustedCertEntry,
cm-r01nn01.mws.mds.xyz, Apr 14, 2019, trustedCertEntry,
[root@cm-r01en01 pki]#
[ Name Node / CM Server]
[root@cm-r01nn01 pki]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts|grep cm
Enter keystore password: changeit
acraizfnmt-rcm, Mar 26, 2019, trustedCertEntry,
cm-r01nn02.mws.mds.xyz, Apr 14, 2019, trustedCertEntry,
cm-c01.mws.mds.xyz, Jul 6, 2019, trustedCertEntry,
cm-r01nn01.mws.mds.xyz, Mar 31, 2019, trustedCertEntry,
[root@cm-r01nn01 pki]#
The only other message that I got when moving certs around under the agent.pem symlink, was:
SSLError: sslv3 alert bad certificate
I'm not clear if that error is a step forward or backwards. Also, discovered by accident that I may not need to restart the cloudera-scm-agent under every scenario. Just changing where the symlink agent.pem points too appear to generate different messages, even when the agent is running.
Thx,
TK
