Steps to fix Ambari-server & agent expired certs

[#1]

Usually Ambari server generates certs with 1 year validity. after an year all Agent would fail to communicate with Ambari-server. Agent and Server certs would be expired. below steps can be followed to replace/resolve the expired certs.

1. stop ambari-server

2. take a back of existing /var/lib/ambari-server/keys folder and empty it.

3. download the attached keys.zip file and copy it to /var/lib/ambari-server/ , your new folder structure should be like /var/lib/ambari-server/keys/ca.config,/var/lib/ambari-server/keys/db/,  - basically this is a fresh keys folder ( this is what you get when you install ambari-server )

4. Take a back up of all the Agent certs located at /var/lib/ambari-agent/keys/ in all the hosts.

5. Delete all the files under /var/lib/ambari-agent/keys/ folder

6. restart ambari-server.
Note: ambari-server should create new certs under /var/lib/ambari-server/keys/ca.crt , /var/lib/ambari-server/keys/ca.key ....

7. restart ambari-agent
Note: ambari-agent should create new certs under /var/lib/ambari-server/keys/ folder

now you should see the successful heart beat from all the Agents.

Note: If Encryption is enabled on Ambari - copy back credentials.jceks, master files from the backed up keys to newly created keys folder. 

Note: Please note that if SSL is enabled for Ambari UI then have to re-enable SSL step again as some of the certs were not part of the keys folder. or else those files can be copied to new keys folders.

[#2]

@apappu, Steps 1 through 3 are only needed if the CA cert generated by Ambari is expired. If it is not expired, then only steps 4, 5, and 7 should be needed.

Did you verify that the Ambari-generated CA cert was expired.

[#3]

@Robert Levas

Yes - that is correct, I could see that server cert expiry also is 365 days set during the creation - hence most likely server cert also will get expire.

----

openssl ca -create_serial -out /var/lib/ambari-server/keys/ca.crt -days 365 -keyfile /var/lib/ambari-server/keys/ca.key -key **** -selfsign -extensions jdk7_ca -config /var/lib/ambari-server/keys/ca.config -batch -infiles /var/lib/ambari-server/keys/ca.csr

[#4]

You are correct. It appears that by default Ambari's CA cert is value for 365 days.

See https://github.com/apache/ambari/blob/2ad42074f1633c5c6f56cf979bdaa49440457566/ambari-server/src/mai...

private static final String SIGN_SRVR_CRT = "openssl ca -create_serial " +
  "-out {1}" + File.separator + "{3} -days 365 -keyfile {1}" + File.separator + "{2} -key {0} -selfsign " +
  "-extensions jdk7_ca -config {1}" + File.separator + "ca.config -batch " +
  "-infiles {1}" + File.separator + "{5}";

Notice: -days 365

[#5]

You are correct. It appears that by default Ambari's CA cert is value for 365 days.

See https://github.com/apache/ambari/blob/2ad42074f1633c5c6f56cf979bdaa49440457566/ambari-server/src/mai...

private static final String SIGN_SRVR_CRT = "openssl ca -create_serial " +
  "-out {1}" + File.separator + "{3} -days 365 -keyfile {1}" + File.separator + "{2} -key {0} -selfsign " +
  "-extensions jdk7_ca -config {1}" + File.separator + "ca.config -batch " +
  "-infiles {1}" + File.separator + "{5}";

Notice: -days 365