How to: Ambari server two-way SSL using custom CA signed certs.

[#1]

Pre-requisites:

1. You need to have the correct passwords
2. The PrivateKey of the CA is required to sign off all the agent certificates.
3. Password for the pass.txt file will be present in clear-text format. And currently, there is no way to encrypt it.

Steps:

1. Back up and delete all the crt and csr files that start with hostname at /var/lib/ambari-server/keys.

   tar -cvzf keys_directory.tar.gz /var/lib/ambari-server/keys/*

2. Empty /var/lib/ambari-server/keys/db/index.txt file

   mv /var/lib/ambari-server/keys/db/index.txt /tmp

3. Delete any certificates under /var/lib/ambari-server/keys/db/newcerts/

   mv /var/lib/ambari-server/keys/db/newcerts/* /tmp

4. Configure CA-signed certificate artifacts in Ambari Server by:
5. Copy the CA-signed certificate, key files to /var/lib/ambari-server/keys/ e.g. rootca.crt, root_server.key
6. Create PKCS keystore file from your certificate and key files.

   openssl pkcs12 -export -in rootca.crt -inkey /var/lib/ambari-server/keys/root_server.key -certfile rootca.crt -out keystore_CA.p12 -password pass:hadoop1120 -passin pass:hadoop1120

7. Create a text file with the appropriate password that has been provided for the keystore in /var/lib/ambari-server/keys directory:

   echo hadoop1120 > pass_CA.txt

8. Next, configure Ambari to use the new cert, keys, and keystore files.
9. Edit the /etc/ambari-server/conf/ambari.properties and set the following keys and values:

   security.server.cert_name=rootca.crt  
   security.server.key_name=root_server.key  
   security.server.keystore_name=keystore_CA.p12  
   security.server.truststore_name=keystore_CA.p12  
   security.server.crt_pass_file=pass_CA.txt  
   security.server.two_way_ssl=true

10. After adding the properties, clean up existing Ambari agent keys information by removing all files in the /var/lib/ambari-agent/keys/ directory on each node that has Ambari agent installed.

Note: Ambari server uses the following command to generate the certificates using the configs above. This command requires the ca.config to be present at the location. (Content of ca.config does not matter):

openssl ca -config /var/lib/ambari-server/keys/ca.config -in /var/lib/ambari-server/keys/node1.coelab.cloudera.com.csr -out /var/lib/ambari-server/keys/node1.coelab.cloudera.com.crt -batch -passin pass:**** -keyfile /var/lib/ambari-server/keys/root_server.key -cert /var/lib/ambari-server/keys/rootca.crt

openssl ca -config /var/lib/ambari-server/keys/ca.config -in /var/lib/ambari-server/keys/node2.coelab.cloudera.com.csr -out /var/lib/ambari-server/keys/node2.coelab.cloudera.com.crt -batch -passin pass:**** -keyfile /var/lib/ambari-server/keys/root_server.key -cert /var/lib/ambari-server/keys/rootca.crt

openssl ca -config /var/lib/ambari-server/keys/ca.config -in /var/lib/ambari-server/keys/c2218-node4.coelab.cloudera.com.csr -out /var/lib/ambari-server/keys/c2218-node4.coelab.cloudera.com.crt -batch -passin pass:**** -keyfile /var/lib/ambari-server/keys/root_server.key -cert /var/lib/ambari-server/keys/rootca.crt

openssl ca -config /var/lib/ambari-server/keys/ca.config -in /var/lib/ambari-server/keys/hostname.cloudera.com.csr -out /var/lib/ambari-server/keys/hostname.cloudera.com.crt -batch -passin pass:**** -keyfile /var/lib/ambari-server/keys/root_server.key -cert /var/lib/ambari-server/keys/rootca.crt

Sample logs from ambari-server.log file after implementation:

[root@node1 ambari-server]# grep "Validating agent hostname:" ambari-server.log -A3

2021-09-22 10:52:46,862  INFO [qtp-ambari-agent-222] CertificateManager:238 - Validating agent hostname: hostname.cloudera.com
2021-09-22 10:52:46,862  INFO [qtp-ambari-agent-222] CertificateManager:250 - Verifying passphrase
2021-09-22 10:52:46,891  INFO [qtp-ambari-agent-222] ShellCommandUtil:63 - Command openssl ca -config /var/lib/ambari-server/keys/ca.config -in /var/lib/ambari-server/keys/hostname.cloudera.com.csr -out /var/lib/ambari-server/keys/hostname.cloudera.com.crt -batch -passin pass:**** -keyfile /var/lib/ambari-server/keys/rootca_PKey.key -cert /var/lib/ambari-server/keys/rootca.crt was finished with exit code: 0 - the operation was completely successfully.
2021-09-22 10:52:47,522  INFO [agent-register-processor-0] HeartBeatHandler:321 - agentOsType = centos7
--
2021-09-22 10:52:51,929  INFO [qtp-ambari-agent-222] CertificateManager:238 - Validating agent hostname: node4.coelab.cloudera.com
2021-09-22 10:52:51,929  INFO [qtp-ambari-agent-222] CertificateManager:250 - Verifying passphrase
2021-09-22 10:52:51,954  INFO [qtp-ambari-agent-222] ShellCommandUtil:63 - Command openssl ca -config /var/lib/ambari-server/keys/ca.config -in /var/lib/ambari-server/keys/hostname.coelab.cloudera.com.csr -out /var/lib/ambari-server/keys/hostname.cloudera.com.crt -batch -passin pass:**** -keyfile /var/lib/ambari-server/keys/rootca_PKey.key -cert /var/lib/ambari-server/keys/rootca.crt was finished with exit code: 0 - the operation was completely successfully.
2021-09-22 10:52:52,450  INFO [agent-register-processor-1] HeartBeatHandler:321 - agentOsType = centos7
--
2021-09-22 10:52:53,227  INFO [qtp-ambari-agent-222] CertificateManager:238 - Validating agent hostname: hostname.coelab.cloudera.com
2021-09-22 10:52:53,227  INFO [qtp-ambari-agent-222] CertificateManager:250 - Verifying passphrase
2021-09-22 10:52:53,253  INFO [qtp-ambari-agent-222] ShellCommandUtil:63 - Command openssl ca -config /var/lib/ambari-server/keys/ca.config -in /var/lib/ambari-server/keys/hostname.cloudera.com.csr -out /var/lib/ambari-server/keys/hostname.cloudera.com.crt -batch -passin pass:**** -keyfile /var/lib/ambari-server/keys/rootca_PKey.key -cert /var/lib/ambari-server/keys/rootca.crt was finished with exit code: 0 - the operation was completely successfully.
2021-09-22 10:52:53,683  INFO [agent-register-processor-2] HeartBeatHandler:321 - agentOsType = centos7

Sample logs after successful registration of the ambari-agent with the ambari-server from the ambari-agent.log:

INFO 2021-09-22 10:52:52,880 NetUtil.py:61 - Connecting to https://hostname.cloudera.com:8440/connection_info
INFO 2021-09-22 10:52:52,952 security.py:59 - Server require two-way SSL authentication. Use it instead of one-way...
INFO 2021-09-22 10:52:52,952 security.py:61 - Connecting to wss://hostname.cloudera.com:8441/agent/stomp/v1
INFO 2021-09-22 10:52:52,952 security.py:221 - Server certicate not exists, downloading
INFO 2021-09-22 10:52:52,953 security.py:244 - Downloading server cert from https://c2218-node1.coelab.cloudera.com:8440/cert/ca/
INFO 2021-09-22 10:52:53,019 security.py:229 - Agent key not exists, generating request
INFO 2021-09-22 10:52:53,020 security.py:301 - openssl req -new -newkey rsa -nodes -keyout "/var/lib/ambari-agent/keys/hostname.com.key" -subj /OU=hostname.cloudera.com/ -out "/var/lib/ambari-agent/keys/hostname.com.csr"
INFO 2021-09-22 10:52:53,158 security.py:237 - Agent certificate not exists, sending sign request
INFO 2021-09-22 10:52:53,349 transport.py:329 - Starting receiver loop
INFO 2021-09-22 10:52:53,352 security.py:88 - SSL connection established. Two-way SSL authentication completed successfully.
INFO 2021-09-22 10:52:53,674 HeartbeatThread.py:127 - Sending registration request