Adding KDC Administrator Credentials to the Ambari Credential Store

[#1]

When enabling Kerberos, where Ambari is managing the Kerberos identities (principals and keytab files), the KDC administrator's credentials are needed in order to create the needed accounts in the KDC.

During the process of enabling Kerberos via Ambari's web-based UI, the user is prompted for this information and has the option to store the credentials in either the temporary or persisted credential store. The temporary credential store is a keystore in memory where each entry is removed after 90 minutes (from initial creation), when Ambari is restarted, or by user request. The persisted credential store is a keystore stored on disk where each entry is removed only by user request. The option to store a credential in the persisted store is only available if Ambari's credential store has been setup.

Creating Ambari's Credential Store

To set up Ambari's credential store, the following commands must be invoked from the Ambari server host's command line:

ambari-server setup-security

Option #2, "Encrypt passwords stored in ambari.properties file", is then used to start the process:

Example:

ambari-server setup-security
Using python  /usr/bin/python
Security setup options...
===========================================================================
Choose one of the following options:
  [1] Enable HTTPS for Ambari server.
  [2] Encrypt passwords stored in ambari.properties file.
  [3] Setup Ambari kerberos JAAS configuration.
  [4] Setup truststore.
  [5] Import certificate to truststore.
===========================================================================
Enter choice, (1-5): 2
Please provide master key for locking the credential store:
Re-enter master key:
Do you want to persist master key. If you choose not to persist, you need to provide the Master Key while starting the ambari server as an env variable named AMBARI_SECURITY_MASTER_KEY or the start will prompt for the master key. Persist [y/n] (y)? y
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup-security' completed successfully.

Once this is complete, the Ambari credential store will be located at /var/lib/ambari-server/keys/credentials.jceks.

To test the password entered when creating the credential store, you can issue the following command:

$JAVA_HOME/bin/keytool -list \
-keystore /var/lib/ambari-server/keys/credentials.jceks \
-storetype JCEKS

Where $JAVA_HOME is the path to where Java is installed. For example, /usr/jdk64/jdk1.8.0_60.

Example:

/usr/jdk64/jdk1.8.0_60/bin/keytool -list \
> -keystore /var/lib/ambari-server/keys/credentials.jceks \
> -storetype JCEKS
Enter keystore password:
Keystore type: JCEKS
Keystore provider: SunJCE
Your keystore contains 1 entry
ambari.db.password, Jul 1, 2016, SecretKeyEntry,

Note that this is optional if there is no need to persist the KDC administrator credentials. However, if opting to create Ambari's credential store, other passwords typically stored in plaintext in Ambari's properties file (/etc/ambari/conf/ambari-server.properties) will be encrypted and stored in the created credential store.

Restarting Ambari

After this has been set up, Ambari must be restarted in order for it to acquire the new information about the credentials store. If Ambari was stopped before setting up the credential store, it must be started.

ambari-server start

If Ambari was not stopped before setting up the credential store, it must be restarted.

ambari-server restart

Storing the KDC Administrator's Credential

From outside Ambari's web-based user interface, there are two ways to manually add the KDC administrator's credential to one of Ambari's credentials stores: via Ambari's REST API and via the Java keytool command.

To add a credential via Ambari's REST API, a cluster must already be established. This is because credentials are linked to clusters to avoid collisions in the event Ambari is able to manage multiple clusters.

The URL for the credentials entrypoint is

/api/v1/clusters/:CLUSTER_NAME/credentials

where :CLUSTER_NAME is the name of the relevant cluster.

This entrypoint can be used to add, get, update, and remove credentials. However, when getting credentials via this interface, no secret information will be return.

To set the KDC administrator's credential, the following information must be supplied:

• KDC administrator principal
• KDC administrator key (or password)
• credential store type ["persisted" or "temporary"]

Also, the credential's alias must be appended to the entrypoint and must be "kdc.admin.credential"

Example:

POST /api/v1/clusters/:CLUSTER_NAME/credentials/kdc.admin.credential

{ 
  "Credential" : 
  {    
    "principal" : "admin/admin@EXAMPLE.COM",
    "key" : "h4d00p&!",   
    "type" : "persisted"
  }
}

Example using curl:

curl -H "X-Requested-By:ambari" -u admin:admin -X  POST -d '{ "Credential" : { "principal" : "admin/admin@EXAMPLE.COM", "key" : "h4d00p&!", "type" : "persisted" } }' http://ambari.example.com:8080/api/v1/clusters/c1/credentials/kdc.admin.credential

This credential may be retrieved (no matter what store it is in - temporary or persisted) using

GET /api/v1/clusters/:CLUSTER_NAME/credentials/kdc.admin.credential

Example:

curl -H "X-Requested-By:ambari" -u admin:admin -X GET http://ambari.example.com:8080/api/v1/clusters/c1/credentials/kdc.admin.credential

{
  "href" : "http://localhost:8080/api/v1/clusters/c1/credentials/kdc.admin.credential",
  "Credential" : {
    "alias" : "kdc.admin.credential",
    "cluster_name" : "c1",
    "type" : "persisted"
  }
}

To update an existing credential:

PUT /api/v1/clusters/:CLUSTER_NAME/credentials/kdc.admin.credential

Example:

curl -H "X-Requested-By:ambari" -u admin:admin -X PUT -d '{ "Credential" : { "principal" : "admin/admin@EXAMPLE.COM", "key" : "h4d00p&!", "type" : "persisted" } }' http://ambari.example.com:8080/api/v1/clusters/c1/credentials/kdc.admin.credential

It should be noted that the credential can be moved from the persisted store to the temporary store (and vice versa) using this mechanism; however the principal and key values must be supplied. If only updating the key, only the key value is needed.

To remove the KDC administrator credential (from no matter what store it is in - temporary or persisted)

DELETE /api/v1/clusters/:CLUSTER_NAME/credentials/kdc.admin.credential

Example:

curl -H "X-Requested-By:ambari" -u admin:admin -X DELETE http://ambari.example.com:8080/api/v1/clusters/c1/credentials/kdc.admin.credential

To add a credential using the Java keytool command, a cluster does not already need to be established, but the name of the cluster to assign the credential to must be known. Also, Ambari's credential store must have been previously set up and this method only allows for storage into Ambari persisted credential store.

This is useful when preparing for a Kerberized cluster to be created via Blueprints and the KDC administrator password is not to be placed in plaintext in either the Blueprint or Cluster Creation files.

The Java keytool command is typically found in the bin directory under the Java home directory ($JAVA_HOME):

$JAVA_HOME/bin/keytool

For example:

/usr/jdk64/jdk1.8.0_60/bin/keytool

This command can be used to mange keystores like Ambari's credential store; however care must be taken to not destroy any useful information stored in it that Ambari needs to use - like the Ambari DB password.

To set the KDC administrator's credential, the following information must be supplied:

• location of credential store - "/var/lib/ambari-server/keys/credentials.jceks"
• type of credential store - "JCEKS" (Java Cryptography Extension KeyStore)
• password for the credential store (previously created)
• cluster name (existing or to be created)
• KDC administrator principal
• KDC administrator key (password)
• credential alias - "kdc.admin.credential"

The following Java keytool "importpass" command is used to insert the credential into the credential store:

$JAVA_HOME/bin/keytool -importpass \
-keystore /var/lib/ambari-server/keys/credentials.jceks \
-storetype JCEKS \
-alias cluster.CLUSTER_NAME.kdc.admin.credential

Where CLUSTER_NAME is the name of the cluster to link this credential to.

Once executed, the credential store password will be requested; and then if successful, the password (or credential) to store will be requested. If still successful, a prompt for a password to encrypt the new credential will be displayed. Nothing should be entered to use the "same as keystore password".

The credential to store must be in the following format:

PrincipalKeyCredential{"principal":"PRINCIPAL","key":[PASSWORD_CHARS]}

Where, PRINCIPAL is the KDC administrator's principal and PASSWORD_CHARS are the characters of the password spilt into a comma-delimited array of quoted characters. For example, "hadoop" becomes "h","a","d","o","o","p"

Example where the KDC administrator principal is "admin/admin@EXAMPLE.COM" and the password is "hadoop":

PrincipalKeyCredential{"principal":"admin/admin@EXAMPLE.COM","key":["h","a","d","o","o","p"]}

Example:

/usr/jdk64/jdk1.8.0_60/bin/keytool -importpass \
> -keystore /var/lib/ambari-server/keys/credentials.jceks \
> -storetype JCEKS \
> -alias cluster.c1.kdc.admin.credential
Enter keystore password:
Enter the password to be stored:
Re-enter password:
Enter key password for <cluster.c1.kdc.admin.credential>
	(RETURN if same as keystore password):

To remove the KDC administrator credential from the credential store, use the "delete" command:

/usr/jdk64/jdk1.8.0_60/bin/keytool -delete \
> -keystore /var/lib/ambari-server/keys/credentials.jceks \
> -storetype JCEKS \
> -alias cluster.c1.kdc.admin.credential

Updates are not possible using the Java keytool command; however the credential may be deleted and re-added.

For more information about using Java keytool, see Oracle's keytool reference page

[#2]

@Robert Levas Can you add "ambari-server restart" before Storing the KDC Administrator's Credential?

Correct me if i am wrong.

[#3]

Hi Robert,

Can we add steps to stop Ambari server before running ambari-server setup-security and start ambari after selecting option 2?

Without restarting Ambari-server, adding KDC admin principal using API method would fail again.

[#4]

Added, thanks for the suggestion.

[#5]

Added, thanks for the suggestion.