When enabling Kerberos, where Ambari is managing the Kerberos identities (principals and keytab files), the KDC administrator's credentials are needed in order to create the needed accounts in the KDC. During the process of enabling Kerberos via Ambari's web-based UI, the user is prompted for this information and has the option to store the credentials in either the temporary or persisted credential store. The temporary credential store is a keystore in memory where each entry is removed after 90 minutes (from initial creation), when Ambari is restarted, or by user request. The persisted credential store is a keystore stored on disk where each entry is removed only by user request. The option to store a credential in the persisted store is only available if Ambari's credential store has been setup. Creating Ambari's Credential Store To set up Ambari's credential store, the following commands must be invoked from the Ambari server host's command line: ambari-server setup-security Option #2, "Encrypt passwords stored in ambari.properties file", is then used to start the process: Example: ambari-server setup-security Using python /usr/bin/python Security setup options... =========================================================================== Choose one of the following options: [1] Enable HTTPS for Ambari server. [2] Encrypt passwords stored in ambari.properties file. [3] Setup Ambari kerberos JAAS configuration. [4] Setup truststore. [5] Import certificate to truststore. =========================================================================== Enter choice, (1-5): 2 Please provide master key for locking the credential store: Re-enter master key: Do you want to persist master key. If you choose not to persist, you need to provide the Master Key while starting the ambari server as an env variable named AMBARI_SECURITY_MASTER_KEY or the start will prompt for the master key. Persist [y/n] (y)? y Adjusting ambari-server permissions and ownership... Ambari Server 'setup-security' completed successfully. Once this is complete, the Ambari credential store will be located at /var/lib/ambari-server/keys/credentials.jceks. To test the password entered when creating the credential store, you can issue the following command: $JAVA_HOME/bin/keytool -list \ -keystore /var/lib/ambari-server/keys/credentials.jceks \ -storetype JCEKS Where $JAVA_HOME is the path to where Java is installed. For example, /usr/jdk64/jdk1.8.0_60. Example: /usr/jdk64/jdk1.8.0_60/bin/keytool -list \ > -keystore /var/lib/ambari-server/keys/credentials.jceks \ > -storetype JCEKS Enter keystore password: Keystore type: JCEKS Keystore provider: SunJCE Your keystore contains 1 entry ambari.db.password, Jul 1, 2016, SecretKeyEntry, Note that this is optional if there is no need to persist the KDC administrator credentials. However, if opting to create Ambari's credential store, other passwords typically stored in plaintext in Ambari's properties file (/etc/ambari/conf/ambari-server.properties) will be encrypted and stored in the created credential store. Restarting Ambari After this has been set up, Ambari must be restarted in order for it to acquire the new information about the credentials store. If Ambari was stopped before setting up the credential store, it must be started. ambari-server start If Ambari was not stopped before setting up the credential store, it must be restarted. ambari-server restart Storing the KDC Administrator's Credential From outside Ambari's web-based user interface, there are two ways to manually add the KDC administrator's credential to one of Ambari's credentials stores: via Ambari's REST API and via the Java keytool command. To add a credential via Ambari's REST API, a cluster must already be established. This is because credentials are linked to clusters to avoid collisions in the event Ambari is able to manage multiple clusters. The URL for the credentials entrypoint is /api/v1/clusters/:CLUSTER_NAME/credentials where :CLUSTER_NAME is the name of the relevant cluster. This entrypoint can be used to add, get, update, and remove credentials. However, when getting credentials via this interface, no secret information will be return. To set the KDC administrator's credential, the following information must be supplied: KDC administrator principal KDC administrator key (or password) credential store type ["persisted" or "temporary"] Also, the credential's alias must be appended to the entrypoint and must be "kdc.admin.credential" Example: POST /api/v1/clusters/:CLUSTER_NAME/credentials/kdc.admin.credential { "Credential" : { "principal" : "admin/admin@EXAMPLE.COM", "key" : "h4d00p&!", "type" : "persisted" } } Example using curl: curl -H "X-Requested-By:ambari" -u admin:admin -X POST -d '{ "Credential" : { "principal" : "admin/admin@EXAMPLE.COM", "key" : "h4d00p&!", "type" : "persisted" } }' http://ambari.example.com:8080/api/v1/clusters/c1/credentials/kdc.admin.credential This credential may be retrieved (no matter what store it is in - temporary or persisted) using GET /api/v1/clusters/:CLUSTER_NAME/credentials/kdc.admin.credential Example: curl -H "X-Requested-By:ambari" -u admin:admin -X GET http://ambari.example.com:8080/api/v1/clusters/c1/credentials/kdc.admin.credential { "href" : "http://localhost:8080/api/v1/clusters/c1/credentials/kdc.admin.credential", "Credential" : { "alias" : "kdc.admin.credential", "cluster_name" : "c1", "type" : "persisted" } } To update an existing credential: PUT /api/v1/clusters/:CLUSTER_NAME/credentials/kdc.admin.credential Example: curl -H "X-Requested-By:ambari" -u admin:admin -X PUT -d '{ "Credential" : { "principal" : "admin/admin@EXAMPLE.COM", "key" : "h4d00p&!", "type" : "persisted" } }' http://ambari.example.com:8080/api/v1/clusters/c1/credentials/kdc.admin.credential It should be noted that the credential can be moved from the persisted store to the temporary store (and vice versa) using this mechanism; however the principal and key values must be supplied. If only updating the key, only the key value is needed. To remove the KDC administrator credential (from no matter what store it is in - temporary or persisted) DELETE /api/v1/clusters/:CLUSTER_NAME/credentials/kdc.admin.credential Example: curl -H "X-Requested-By:ambari" -u admin:admin -X DELETE http://ambari.example.com:8080/api/v1/clusters/c1/credentials/kdc.admin.credential To add a credential using the Java keytool command, a cluster does not already need to be established, but the name of the cluster to assign the credential to must be known. Also, Ambari's credential store must have been previously set up and this method only allows for storage into Ambari persisted credential store. This is useful when preparing for a Kerberized cluster to be created via Blueprints and the KDC administrator password is not to be placed in plaintext in either the Blueprint or Cluster Creation files. The Java keytool command is typically found in the bin directory under the Java home directory ($JAVA_HOME): $JAVA_HOME/bin/keytool For example: /usr/jdk64/jdk1.8.0_60/bin/keytool This command can be used to mange keystores like Ambari's credential store; however care must be taken to not destroy any useful information stored in it that Ambari needs to use - like the Ambari DB password. To set the KDC administrator's credential, the following information must be supplied: location of credential store - "/var/lib/ambari-server/keys/credentials.jceks" type of credential store - "JCEKS" (Java Cryptography Extension KeyStore) password for the credential store (previously created) cluster name (existing or to be created) KDC administrator principal KDC administrator key (password) credential alias - "kdc.admin.credential" The following Java keytool "importpass" command is used to insert the credential into the credential store: $JAVA_HOME/bin/keytool -importpass \ -keystore /var/lib/ambari-server/keys/credentials.jceks \ -storetype JCEKS \ -alias cluster.CLUSTER_NAME.kdc.admin.credential Where CLUSTER_NAME is the name of the cluster to link this credential to. Once executed, the credential store password will be requested; and then if successful, the password (or credential) to store will be requested. If still successful, a prompt for a password to encrypt the new credential will be displayed. Nothing should be entered to use the "same as keystore password". The credential to store must be in the following format: PrincipalKeyCredential{"principal":"PRINCIPAL","key":[PASSWORD_CHARS]} Where, PRINCIPAL is the KDC administrator's principal and PASSWORD_CHARS are the characters of the password spilt into a comma-delimited array of quoted characters. For example, "hadoop" becomes "h","a","d","o","o","p" Example where the KDC administrator principal is "admin/admin@EXAMPLE.COM" and the password is "hadoop": PrincipalKeyCredential{"principal":"admin/admin@EXAMPLE.COM","key":["h","a","d","o","o","p"]} Example: /usr/jdk64/jdk1.8.0_60/bin/keytool -importpass \ > -keystore /var/lib/ambari-server/keys/credentials.jceks \ > -storetype JCEKS \ > -alias cluster.c1.kdc.admin.credential Enter keystore password: Enter the password to be stored: Re-enter password: Enter key password for <cluster.c1.kdc.admin.credential> (RETURN if same as keystore password): To remove the KDC administrator credential from the credential store, use the "delete" command: /usr/jdk64/jdk1.8.0_60/bin/keytool -delete \ > -keystore /var/lib/ambari-server/keys/credentials.jceks \ > -storetype JCEKS \ > -alias cluster.c1.kdc.admin.credential Updates are not possible using the Java keytool command; however the credential may be deleted and re-added. For more information about using Java keytool, see Oracle's keytool reference page ... View more

Once Kerberos is enabled, it is possible to get a listing of the expected Kerberos principals and keytab files. This data is typically used when manually managing the Kerberos Identities for the Ambari cluster and may be easily downloaded as a CSV file from the Enable Kerberos Wizard in the Ambari UI. After Kerberos is enabled in Ambari, details about the expected Kerberos Identities may be obtained using the following REST API call: GET /api/v1/clusters/:cluster_name/kerberos_identities Note: replace :cluster_name with the name of your cluster. The result of this query is a JSON formatted document containing a high level listing of the expected identities. { "href" : "http://ambari-server:8080/api/v1/clusters/c1/kerberos_identities", "items" : [ { "href" : "http://ambari-server:8080/api/v1/clusters/c1/kerberos_identities/HTTP%2Fhost-1.example.com%40EXAMPLE.COM", "KerberosIdentity" : { "cluster_name" : "c1", "host_name" : "host-1.example.com", "principal_name" : "HTTP/host-1.example.com@EXAMPLE.COM" } }, { "href" : "http://ambari-server:8080/api/v1/clusters/c1/kerberos_identities/ambari-qa-c1%40EXAMPLE.COM", "KerberosIdentity" : { "cluster_name" : "c1", "host_name" : "host-1.example.com", "principal_name" : "ambari-qa-c1@EXAMPLE.COM" } }, { "href" : "http://ambari-server:8080/api/v1/clusters/c1/kerberos_identities/dn%2Fhost-1.example.com%40EXAMPLE.COM", "KerberosIdentity" : { "cluster_name" : "c1", "host_name" : "host-1.example.com", "principal_name" : "dn/host-1.example.com@EXAMPLE.COM" } }, { "href" : "http://ambari-server:8080/api/v1/clusters/c1/kerberos_identities/hdfs-c1%40EXAMPLE.COM", "KerberosIdentity" : { "cluster_name" : "c1", "host_name" : "host-1.example.com", "principal_name" : "hdfs-c1@EXAMPLE.COM" } }, ... To get more information on each identity follow the provided URLs in the output. For example, for ambari-qa-c1@EXAMPLE.COM the URL would be: http://ambari-server:8080/api/v1/clusters/c1/kerberos_identities/ambari-qa-c1%40EXAMPLE.COM Which will yield something like: [ { "href" : "http://ambari-server:8080/api/v1/clusters/c1/kerberos_identities/ambari-qa-c1%40EXAMPLE.COM", "KerberosIdentity" : { "cluster_name" : "c1", "description" : "/smokeuser", "host_name" : "host-1.example.com", "keytab_file_group" : "hadoop", "keytab_file_group_access" : "r", "keytab_file_installed" : "true", "keytab_file_mode" : "440", "keytab_file_owner" : "ambari-qa", "keytab_file_owner_access" : "r", "keytab_file_path" : "/etc/security/keytabs/smokeuser.headless.keytab", "principal_local_username" : "ambari-qa", "principal_name" : "ambari-qa-c1@EXAMPLE.COM", "principal_type" : "USER" } }, { "href" : "http://ambari-server:8080/api/v1/clusters/c1/kerberos_identities/ambari-qa-c1%40EXAMPLE.COM", "KerberosIdentity" : { "cluster_name" : "c1", "description" : "/smokeuser", "host_name" : "host-2.example.com", "keytab_file_group" : "hadoop", "keytab_file_group_access" : "r", "keytab_file_installed" : "true", "keytab_file_mode" : "440", "keytab_file_owner" : "ambari-qa", "keytab_file_owner_access" : "r", "keytab_file_path" : "/etc/security/keytabs/smokeuser.headless.keytab", "principal_local_username" : "ambari-qa", "principal_name" : "ambari-qa-c1@EXAMPLE.COM", "principal_type" : "USER" } }, ... To get the all of the data in one query, indicate that you want all of the field data by appending fields=* to the original query: GET /api/v1/clusters/:cluster_name/kerberos_identities?fields=* This will yield something like the following: { "href" : "http://ambari-server:8080/api/v1/clusters/c1/kerberos_identities?fields=*", "items" : [ { "href" : "http://ambari-server:8080/api/v1/clusters/c1/kerberos_identities/HTTP%2Fhost-1.example.com%40EXAMPLE.COM", "KerberosIdentity" : { "cluster_name" : "c1", "description" : "/spnego", "host_name" : "host-1.example.com", "keytab_file_group" : "hadoop", "keytab_file_group_access" : "r", "keytab_file_installed" : "true", "keytab_file_mode" : "440", "keytab_file_owner" : "root", "keytab_file_owner_access" : "r", "keytab_file_path" : "/etc/security/keytabs/spnego.service.keytab", "principal_local_username" : null, "principal_name" : "HTTP/host-1.example.com@EXAMPLE.COM", "principal_type" : "SERVICE" } }, { "href" : "http://ambari-server:8080/api/v1/clusters/c1/kerberos_identities/ambari-qa-c1%40EXAMPLE.COM", "KerberosIdentity" : { "cluster_name" : "c1", "description" : "/smokeuser", "host_name" : "host-1.example.com", "keytab_file_group" : "hadoop", "keytab_file_group_access" : "r", "keytab_file_installed" : "true", "keytab_file_mode" : "440", "keytab_file_owner" : "ambari-qa", "keytab_file_owner_access" : "r", "keytab_file_path" : "/etc/security/keytabs/smokeuser.headless.keytab", "principal_local_username" : "ambari-qa", "principal_name" : "ambari-qa-c1@EXAMPLE.COM", "principal_type" : "USER" } }, { "href" : "http://ambari-server:8080/api/v1/clusters/c1/kerberos_identities/dn%2Fhost-1.example.com%40EXAMPLE.COM", "KerberosIdentity" : { "cluster_name" : "c1", "description" : "datanode_dn", "host_name" : "host-1.example.com", "keytab_file_group" : "hadoop", "keytab_file_group_access" : "", "keytab_file_installed" : "true", "keytab_file_mode" : "400", "keytab_file_owner" : "hdfs", "keytab_file_owner_access" : "r", "keytab_file_path" : "/etc/security/keytabs/dn.service.keytab", "principal_local_username" : "hdfs", "principal_name" : "dn/host-1.example.com@EXAMPLE.COM", "principal_type" : "SERVICE" } }, ... In many cases, this may be good enough. However there may be a need to get this data in a different format, especially when using the data in a script. In that case, it is possible to retrieve this as CSV formatted data by appending format=CSV to the query: GET /api/v1/clusters/:cluster_name/kerberos_identities?fields=*&format=CSV The CSV formatted data will look something like: host,description,principal name,principal type,local username,keytab file path,keytab file owner,keytab file owner access,keytab file group,keytab file group access,keytab file mode,keytab file installed host-1.example.com,/spnego,HTTP/host-1.example.com@EXAMPLE.COM,SERVICE,,/etc/security/keytabs/spnego.service.keytab,root,r,hadoop,r,440,true host-1.example.com,/smokeuser,ambari-qa-c1@EXAMPLE.COM,USER,ambari-qa,/etc/security/keytabs/smokeuser.headless.keytab,ambari-qa,r,hadoop,r,440,true host-1.example.com,datanode_dn,dn/host-1.example.com@EXAMPLE.COM,SERVICE,hdfs,/etc/security/keytabs/dn.service.keytab,hdfs,r,hadoop,,400,true ... host-1.example.com,hdfs,hdfs-c1@EXAMPLE.COM,USER,hdfs,/etc/security/keytabs/hdfs.headless.keytab,hdfs,r,hadoop,r,440,true host-1.example.com,namenode_nn,nn/host-1.example.com@EXAMPLE.COM,SERVICE,hdfs,/etc/security/keytabs/nn.service.keytab,hdfs,r,hadoop,,400,true host-1.example.com,zookeeper_zk,zookeeper/host-1.example.com@EXAMPLE.COM,SERVICE,,/etc/security/keytabs/zk.service.keytab,zookeeper,r,hadoop,,400,true host-2.example.com,/spnego,HTTP/host-2.example.com@EXAMPLE.COM,SERVICE,,/etc/security/keytabs/spnego.service.keytab,root,r,hadoop,r,440,true host-2.example.com,/smokeuser,ambari-qa-c1@EXAMPLE.COM,USER,ambari-qa,/etc/security/keytabs/smokeuser.headless.keytab,ambari-qa,r,hadoop,r,440,true host-2.example.com,datanode_dn,dn/host-2.example.com@EXAMPLE.COM,SERVICE,hdfs,/etc/security/keytabs/dn.service.keytab,hdfs,r,hadoop,,400,true host-2.example.com,secondary_namenode_nn,nn/host-2.example.com@EXAMPLE.COM,SERVICE,hdfs,/etc/security/keytabs/nn.service.keytab,hdfs,r,hadoop,,400,true host-2.example.com,zookeeper_zk,zookeeper/host-2.example.com@EXAMPLE.COM,SERVICE,,/etc/security/keytabs/zk.service.keytab,zookeeper,r,hadoop,,400,true ... View more