Recreate Ambari's CA, used to sign SSL certificates for 2-way SSL

[#1]

There are certain circumstances where the Ambari CA needs to be recreated. Maybe it was corrupted, maybe Ambari's CA certificate expired, etc.... This how-to, assumes that the certificates signed by the Ambari CA are replaceable - which is generally the case for certificates used by Ambari agents for 2-way SSL connections.

In the end, the Ambari server and all the agents will be restarted, causing a new CA certificate to be created along with new SSL certificates for each of the Ambari agents.

On the Ambari server:

• Stop the Ambari server
• Backup /var/lib/ambari-server/keys and it child directories
• Delete the following files from /var/lib/ambari-server/keys
  • ca.key
  • ca.csr
  • ca.crt
  • pass.txt
  • keystore.p12
  • *.csr
  • *.crt
• Delete the following files from /var/lib/ambari-server/keys/db
  • index.txt.old
  • index.txt.attr.old
  • serial.old
• Truncate the following files from /var/lib/ambari-server/keys/db
  • index.txt
  • index.txt.attr
• Edit the following files from /var/lib/ambari-server/keys/db
  • serial
    • set the contents to be exactly

      00
      

• Delete all files under /var/lib/ambari-server/keys/db/newcerts
• Restart Ambari server.

On each Ambari agent host:

• Stop the Ambari agent
• Backup /var/lib/ambari-agent/keys and it child directories
• Delete the following files from /var/lib/ambari-agent/keys
  • ca.crt
  • *.crt
  • *.csr
  • *.key
• Restart Ambari agent

After restarting the Ambari server, the following (or similar) entries should be seen in the /var/log/ambari-server/ambari-server.log file:

12 Jun 2017 14:38:19,606  INFO [main] ShellCommandUtil:63 - Command openssl genrsa -des3 -passout pass:**** -out /var/lib/ambari-server/keys/ca.key 4096  was finished with exit code: 0 - the operation was completely successfully
.
12 Jun 2017 14:38:19,640  INFO [main] ShellCommandUtil:63 - Command openssl req -passin pass:**** -new -key /var/lib/ambari-server/keys/ca.key -out /var/lib/ambari-server/keys/ca.csr -batch was finished with exit code: 0 - the o
peration was completely successfully.
12 Jun 2017 14:38:19,683  INFO [main] ShellCommandUtil:63 - Command openssl ca -create_serial -out /var/lib/ambari-server/keys/ca.crt -days 365 -keyfile /var/lib/ambari-server/keys/ca.key -key **** -selfsign -extensions jdk7_ca -config /var/lib/ambari-server/keys/ca.config -batch -infiles /var/lib/ambari-server/keys/ca.csr was finished with exit code: 0 - the operation was completely successfully.
12 Jun 2017 14:38:19,701  INFO [main] ShellCommandUtil:63 - Command openssl pkcs12 -export -in /var/lib/ambari-server/keys/ca.crt -inkey /var/lib/ambari-server/keys/ca.key -certfile /var/lib/ambari-server/keys/ca.crt -out /var/lib/ambari-server/keys/keystore.p12 -password pass:**** -passin pass:****
 was finished with exit code: 0 - the operation was completely successfully.
12 Jun 2017 14:38:19,708  INFO [main] ShellCommandUtil:63 - Command find /var/lib/ambari-server/keys -type f -exec chmod 700 {} + was finished with exit code: 0 - the operation was completely successfully.
12 Jun 2017 14:38:19,708  INFO [main] ShellCommandUtil:63 - Command chmod 600 /var/lib/ambari-server/keys/pass.txt was finished with exit code: 0 - the operation was completely successfully.

....

12 Jun 2017 14:52:53,797  INFO [qtp-ambari-agent-34] CertificateManager:200 - Signing agent certificate
12 Jun 2017 14:52:53,800  INFO [qtp-ambari-agent-34] CertificateManager:220 - Validating agent hostname: c6401.ambari.apache.org
12 Jun 2017 14:52:53,800  INFO [qtp-ambari-agent-34] CertificateManager:232 - Verifying passphrase
12 Jun 2017 14:52:53,849  INFO [qtp-ambari-agent-34] ShellCommandUtil:63 - Command openssl ca -config /var/lib/ambari-server/keys/ca.config -in /var/lib/ambari-server/keys/c6401.ambari.apache.org.csr -out /var/lib/ambari-server/keys/c6401.ambari.apache.org.crt -batch -passin pass:**** -keyfile /var/lib/ambari-server/keys/ca.key -cert /var/lib/ambari-server/keys/ca.crt was finished with exit code: 0 - the operation was completely successfully.

After restarting the Ambari agent, the following (or similar) entires should be seen in the /var/log/ambari-agent/ambari-agent.log file:

INFO 2017-06-12 14:52:53,625 security.py:55 - Server require two-way SSL authentication. Use it instead of one-way...
INFO 2017-06-12 14:52:53,625 security.py:179 - Server certicate not exists, downloading
INFO 2017-06-12 14:52:53,625 security.py:202 - Downloading server cert from https://localhost:8440/cert/ca/
INFO 2017-06-12 14:52:53,693 security.py:187 - Agent key not exists, generating request
INFO 2017-06-12 14:52:53,693 security.py:258 - openssl req -new -newkey rsa:1024 -nodes -keyout "/var/lib/ambari-agent/keys/c6401.ambari.apache.org.key" -subj /OU=c6401.ambari.apache.org/ -out "/var/lib/ambari-agent/keys/c6401.ambari.apache.org.csr"
INFO 2017-06-12 14:52:53,736 security.py:195 - Agent certificate not exists, sending sign request
INFO 2017-06-12 14:52:53,855 security.py:93 - SSL Connect being called.. connecting to the server
INFO 2017-06-12 14:52:53,933 security.py:77 - SSL connection established. Two-way SSL authentication completed successfully.

[#2]

With the Ambari agents would

ambari-agent reset <ambari server hostname>

accomplish the same thing? Or does it do more than clear certificates?

[#3]

I am not exactly sure all of the tasks that the Ambari agent reset operation performs. It seems like it will do a lot more than just clean up the existing certs - if it does that at all. However, it will not perform any of the Ambari server-side tasks. So at lest the Ambari server steps from the article need to take place manually.