There are certain circumstances where the Ambari CA needs to be recreated. Maybe it was corrupted, maybe Ambari's CA certificate expired, etc.... This how-to, assumes that the certificates signed by the Ambari CA are replaceable - which is generally the case for certificates used by Ambari agents for 2-way SSL connections. In the end, the Ambari server and all the agents will be restarted, causing a new CA certificate to be created along with new SSL certificates for each of the Ambari agents. On the Ambari server: Stop the Ambari server Backup /var/lib/ambari-server/keys and it child directories Delete the following files from /var/lib/ambari-server/keys ca.key ca.csr ca.crt pass.txt keystore.p12 *.csr *.crt Delete the following files from /var/lib/ambari-server/keys/db index.txt.old index.txt.attr.old serial.old Truncate the following files from /var/lib/ambari-server/keys/db index.txt index.txt.attr Edit the following files from /var/lib/ambari-server/keys/db serial set the contents to be exactly 00 Delete all files under /var/lib/ambari-server/keys/db/newcerts Restart Ambari server. On each Ambari agent host: Stop the Ambari agent Backup /var/lib/ambari-agent/keys and it child directories Delete the following files from /var/lib/ambari-agent/keys ca.crt *.crt *.csr *.key Restart Ambari agent After restarting the Ambari server, the following (or similar) entries should be seen in the /var/log/ambari-server/ambari-server.log file: 12 Jun 2017 14:38:19,606 INFO [main] ShellCommandUtil:63 - Command openssl genrsa -des3 -passout pass:**** -out /var/lib/ambari-server/keys/ca.key 4096 was finished with exit code: 0 - the operation was completely successfully . 12 Jun 2017 14:38:19,640 INFO [main] ShellCommandUtil:63 - Command openssl req -passin pass:**** -new -key /var/lib/ambari-server/keys/ca.key -out /var/lib/ambari-server/keys/ca.csr -batch was finished with exit code: 0 - the o peration was completely successfully. 12 Jun 2017 14:38:19,683 INFO [main] ShellCommandUtil:63 - Command openssl ca -create_serial -out /var/lib/ambari-server/keys/ca.crt -days 365 -keyfile /var/lib/ambari-server/keys/ca.key -key **** -selfsign -extensions jdk7_ca -config /var/lib/ambari-server/keys/ca.config -batch -infiles /var/lib/ambari-server/keys/ca.csr was finished with exit code: 0 - the operation was completely successfully. 12 Jun 2017 14:38:19,701 INFO [main] ShellCommandUtil:63 - Command openssl pkcs12 -export -in /var/lib/ambari-server/keys/ca.crt -inkey /var/lib/ambari-server/keys/ca.key -certfile /var/lib/ambari-server/keys/ca.crt -out /var/lib/ambari-server/keys/keystore.p12 -password pass:**** -passin pass:**** was finished with exit code: 0 - the operation was completely successfully. 12 Jun 2017 14:38:19,708 INFO [main] ShellCommandUtil:63 - Command find /var/lib/ambari-server/keys -type f -exec chmod 700 {} + was finished with exit code: 0 - the operation was completely successfully. 12 Jun 2017 14:38:19,708 INFO [main] ShellCommandUtil:63 - Command chmod 600 /var/lib/ambari-server/keys/pass.txt was finished with exit code: 0 - the operation was completely successfully. .... 12 Jun 2017 14:52:53,797 INFO [qtp-ambari-agent-34] CertificateManager:200 - Signing agent certificate 12 Jun 2017 14:52:53,800 INFO [qtp-ambari-agent-34] CertificateManager:220 - Validating agent hostname: c6401.ambari.apache.org 12 Jun 2017 14:52:53,800 INFO [qtp-ambari-agent-34] CertificateManager:232 - Verifying passphrase 12 Jun 2017 14:52:53,849 INFO [qtp-ambari-agent-34] ShellCommandUtil:63 - Command openssl ca -config /var/lib/ambari-server/keys/ca.config -in /var/lib/ambari-server/keys/c6401.ambari.apache.org.csr -out /var/lib/ambari-server/keys/c6401.ambari.apache.org.crt -batch -passin pass:**** -keyfile /var/lib/ambari-server/keys/ca.key -cert /var/lib/ambari-server/keys/ca.crt was finished with exit code: 0 - the operation was completely successfully. After restarting the Ambari agent, the following (or similar) entires should be seen in the /var/log/ambari-agent/ambari-agent.log file: INFO 2017-06-12 14:52:53,625 security.py:55 - Server require two-way SSL authentication. Use it instead of one-way... INFO 2017-06-12 14:52:53,625 security.py:179 - Server certicate not exists, downloading INFO 2017-06-12 14:52:53,625 security.py:202 - Downloading server cert from https://localhost:8440/cert/ca/ INFO 2017-06-12 14:52:53,693 security.py:187 - Agent key not exists, generating request INFO 2017-06-12 14:52:53,693 security.py:258 - openssl req -new -newkey rsa:1024 -nodes -keyout "/var/lib/ambari-agent/keys/c6401.ambari.apache.org.key" -subj /OU=c6401.ambari.apache.org/ -out "/var/lib/ambari-agent/keys/c6401.ambari.apache.org.csr" INFO 2017-06-12 14:52:53,736 security.py:195 - Agent certificate not exists, sending sign request INFO 2017-06-12 14:52:53,855 security.py:93 - SSL Connect being called.. connecting to the server INFO 2017-06-12 14:52:53,933 security.py:77 - SSL connection established. Two-way SSL authentication completed successfully. ... View more

Overview Ambari's Kerberos Descriptor is JSON-formatted document used to help Ambari enable Kerberos for installed services. The descriptor contains the information needed to create the required principals and keytab files. It also declares configuration changes needed by the serviced so they a are configured for Kerberos properly. The Kerberos Descriptor is comprised of the compiled Kerberos descriptors found in the relevant service definitions with user-specified changes applied to it. This combination of data is known as the "Composite Kerberos Descriptor", where the separate parts are known as the "Stack-level Kerberos Descriptor" and the "User-specified Kerberos Descriptor". Each of these descriptors may be obtained from Ambari via is REST API: GET /api/v1/clusters/{CLUSER_NAME}/kerberos_descriptors/COMPOSITE GET /api/v1/clusters/{CLUSER_NAME}/kerberos_descriptors/STACK GET /api/v1/clusters/{CLUSER_NAME}/kerberos_descriptors/USER NOTE: Be sure to replace {CLUSTER_NAME} when the name of the relevant cluster. These REST API calls are for informational purposes only and therefore are read-only. Also, this data is available whether Kerberos is enabled or not. However the User-specified Kerberos Descriptor will most-likely be empty. The Kerberos Descriptor was designed to favor user-supplied change over the stack-level defaults; while maintaining forward compatibility in the even the stack-definitions change by adding new or updating existing component definitions. Because of this, it is expected that the User-specified Kerberos Descriptor is sparse, containing only the changes needed to be applied on top of the stack-level defaults. However as of Ambari 2.4.2, when enabling Kerberos via Ambari's Enable Kerberos Wizard, the complete Kerberos Descriptor is stored as the User-Defined Kerberos Descriptor. Storing the entire Kerberos Descriptor as the User-Defined Kerberos Descriptor is not necessarily a problem since the Composite Kerberos Descriptor will still be valid; and any additions to the Stack-level Kerberos Descriptor will be realized after Ambari server or stack upgrades. Unfortunately, issues can occur when changes to existing pieces of the Stack-level Kerberos Descriptor are encountered durning an upgrade. This is due to ambiguities encountered when upgrading the User-Specified Kerberos descriptor. If an issue with User-Defined Kerberos Descriptor is encountered, it may be necessary to manually edit it. This can be done by Getting the descriptor using Ambari's REST API Editing the descriptor using a text editor Putting the updated descriptor using Ambari's REST API Getting the descriptor using Ambari's REST API To get the User-Specified Kerberos Descriptor, the following REST API call may be issued to Ambari: GET /api/v1/clusters/{CLUSTER_NAME}/artifacts/kerberos_descriptor NOTE: Be sure to replace {CLUSTER_NAME} when the name of the relevant cluster. Notice that the API call access the "artifacts" resource of the cluster rather than "kerberos_descriptors" resource of the cluster. This is due to the storage implementation of the User-Specified Kerberos Descriptor data. If User-Specified Kerberos Descriptor was set, the response will look something like { "href" : "http://host1.example.com:8080/api/v1/clusters/c1/artifacts/kerberos_descriptor", "Artifacts" : { "artifact_name" : "kerberos_descriptor", "cluster_name" : "c1" }, "artifact_data" : { ... } } The user-specified data will exist under the "artifact_data" section, which was removed to brevity. This API call can be issued using a command-line tool like curl and the output can be stored to a local file. For example: curl -u admin:admin -X GET -o kerberos_descriptor.json http://localhost:8080/api/v1/clusters/c1/artifacts/kerberos_descriptor NOTE: The user credentials ("-u admin:admin") and cluster name ("c1") should be changed for the particular cluster. After the call completes, the User-specified Kerberos Descriptor (with some additional metadata) will be in the file named kerberos_descriptor.json in the local directory. Editing the descriptor using a text editor Once the User-specified Kerberos Descriptor has been obtained and stored in a local file, it may be edited using a text editor. Other than any fixes (additions, subtractions, etc...), the following lines in the file must be removed: "href" : "http://host1.example.com:8080/api/v1/clusters/c1/artifacts/kerberos_descriptor", "Artifacts" : { "artifact_name" : "kerberos_descriptor", "cluster_name" : "c1" }, This is metadata that will cause a failure when attempting to store the updated User-specified Kerberos Descriptor. The resulting JSON document should be something like { "artifact_data" : { ... } } The user-specified data will exist under the "artifact_data" section, which was removed to brevity. After all needed changes are made, be sure to save the file. Putting the updated descriptor using Ambari's REST API After the needed changes are made to the User-specified Kerberos Descriptor, it must be stored in Ambari. This is done by issuing the following API call to Ambari while adding the changed data as the payload: PUT /api/v1/clusters/{CLUSTER_NAME}/artifacts/kerberos_descriptor NOTE: Be sure to replace {CLUSTER_NAME} when the name of the relevant cluster. This API call can be issued using a command-line tool like curl and the payload can be specified via a local file. For example: curl -u admin:admin -X PUT -d @kerberos_descriptor.json http://localhost:8080/api/v1/clusters/c1/artifacts/kerberos_descriptor NOTE: The user credentials ("-u admin:admin") and cluster name ("c1") should be changed for the particular cluster. After the call completes, the User-specified Kerberos Descriptor stored in the file named kerberos_descriptor.json will be used to update the stored data in the artifact resource. Ambari should realize the changes without restarting. ... View more