Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
Labels (1)
avatar

There are certain circumstances where the Ambari CA needs to be recreated. Maybe it was corrupted, maybe Ambari's CA certificate expired, etc.... This how-to, assumes that the certificates signed by the Ambari CA are replaceable - which is generally the case for certificates used by Ambari agents for 2-way SSL connections.

In the end, the Ambari server and all the agents will be restarted, causing a new CA certificate to be created along with new SSL certificates for each of the Ambari agents.

On the Ambari server:

  • Stop the Ambari server
  • Backup /var/lib/ambari-server/keys and it child directories
  • Delete the following files from /var/lib/ambari-server/keys
    • ca.key
    • ca.csr
    • ca.crt
    • pass.txt
    • keystore.p12
    • *.csr
    • *.crt
  • Delete the following files from /var/lib/ambari-server/keys/db
    • index.txt.old
    • index.txt.attr.old
    • serial.old
  • Truncate the following files from /var/lib/ambari-server/keys/db
    • index.txt
    • index.txt.attr
  • Edit the following files from /var/lib/ambari-server/keys/db
    • serial
      • set the contents to be exactly
        00
        
  • Delete all files under /var/lib/ambari-server/keys/db/newcerts
  • Restart Ambari server.

On each Ambari agent host:

  • Stop the Ambari agent
  • Backup /var/lib/ambari-agent/keys and it child directories
  • Delete the following files from /var/lib/ambari-agent/keys
    • ca.crt
    • *.crt
    • *.csr
    • *.key
  • Restart Ambari agent

After restarting the Ambari server, the following (or similar) entries should be seen in the /var/log/ambari-server/ambari-server.log file:

12 Jun 2017 14:38:19,606  INFO [main] ShellCommandUtil:63 - Command openssl genrsa -des3 -passout pass:**** -out /var/lib/ambari-server/keys/ca.key 4096  was finished with exit code: 0 - the operation was completely successfully
.
12 Jun 2017 14:38:19,640  INFO [main] ShellCommandUtil:63 - Command openssl req -passin pass:**** -new -key /var/lib/ambari-server/keys/ca.key -out /var/lib/ambari-server/keys/ca.csr -batch was finished with exit code: 0 - the o
peration was completely successfully.
12 Jun 2017 14:38:19,683  INFO [main] ShellCommandUtil:63 - Command openssl ca -create_serial -out /var/lib/ambari-server/keys/ca.crt -days 365 -keyfile /var/lib/ambari-server/keys/ca.key -key **** -selfsign -extensions jdk7_ca -config /var/lib/ambari-server/keys/ca.config -batch -infiles /var/lib/ambari-server/keys/ca.csr was finished with exit code: 0 - the operation was completely successfully.
12 Jun 2017 14:38:19,701  INFO [main] ShellCommandUtil:63 - Command openssl pkcs12 -export -in /var/lib/ambari-server/keys/ca.crt -inkey /var/lib/ambari-server/keys/ca.key -certfile /var/lib/ambari-server/keys/ca.crt -out /var/lib/ambari-server/keys/keystore.p12 -password pass:**** -passin pass:****
 was finished with exit code: 0 - the operation was completely successfully.
12 Jun 2017 14:38:19,708  INFO [main] ShellCommandUtil:63 - Command find /var/lib/ambari-server/keys -type f -exec chmod 700 {} + was finished with exit code: 0 - the operation was completely successfully.
12 Jun 2017 14:38:19,708  INFO [main] ShellCommandUtil:63 - Command chmod 600 /var/lib/ambari-server/keys/pass.txt was finished with exit code: 0 - the operation was completely successfully.

....

12 Jun 2017 14:52:53,797  INFO [qtp-ambari-agent-34] CertificateManager:200 - Signing agent certificate
12 Jun 2017 14:52:53,800  INFO [qtp-ambari-agent-34] CertificateManager:220 - Validating agent hostname: c6401.ambari.apache.org
12 Jun 2017 14:52:53,800  INFO [qtp-ambari-agent-34] CertificateManager:232 - Verifying passphrase
12 Jun 2017 14:52:53,849  INFO [qtp-ambari-agent-34] ShellCommandUtil:63 - Command openssl ca -config /var/lib/ambari-server/keys/ca.config -in /var/lib/ambari-server/keys/c6401.ambari.apache.org.csr -out /var/lib/ambari-server/keys/c6401.ambari.apache.org.crt -batch -passin pass:**** -keyfile /var/lib/ambari-server/keys/ca.key -cert /var/lib/ambari-server/keys/ca.crt was finished with exit code: 0 - the operation was completely successfully.

After restarting the Ambari agent, the following (or similar) entires should be seen in the /var/log/ambari-agent/ambari-agent.log file:

INFO 2017-06-12 14:52:53,625 security.py:55 - Server require two-way SSL authentication. Use it instead of one-way...
INFO 2017-06-12 14:52:53,625 security.py:179 - Server certicate not exists, downloading
INFO 2017-06-12 14:52:53,625 security.py:202 - Downloading server cert from https://localhost:8440/cert/ca/
INFO 2017-06-12 14:52:53,693 security.py:187 - Agent key not exists, generating request
INFO 2017-06-12 14:52:53,693 security.py:258 - openssl req -new -newkey rsa:1024 -nodes -keyout "/var/lib/ambari-agent/keys/c6401.ambari.apache.org.key" -subj /OU=c6401.ambari.apache.org/ -out "/var/lib/ambari-agent/keys/c6401.ambari.apache.org.csr"
INFO 2017-06-12 14:52:53,736 security.py:195 - Agent certificate not exists, sending sign request
INFO 2017-06-12 14:52:53,855 security.py:93 - SSL Connect being called.. connecting to the server
INFO 2017-06-12 14:52:53,933 security.py:77 - SSL connection established. Two-way SSL authentication completed successfully.
3,631 Views
Comments

With the Ambari agents would

ambari-agent reset <ambari server hostname>

accomplish the same thing? Or does it do more than clear certificates?

I am not exactly sure all of the tasks that the Ambari agent reset operation performs. It seems like it will do a lot more than just clean up the existing certs - if it does that at all. However, it will not perform any of the Ambari server-side tasks. So at lest the Ambari server steps from the article need to take place manually.

Version history
Last update:
‎06-12-2017 03:29 PM
Updated by:
Contributors