KMS install failing with client not found in kerberos error

[#1]

if I try to install Ranger KMS in HDP2.5 , I am getting the following error in ambari-server.log

23 Dec 2016 15:17:30,438  INFO [ambari-client-thread-289] AmbariManagementControllerImpl:2329 - AmbariManagementControllerImpl.createHostAction: created ExecutionCommand for host hadoop1.abc.com, role RANGER_KMS_SERVER, roleCommand INSTALL, and command ID 1834--1, with cluster-env tags version1480534831774
23 Dec 2016 15:17:30,452  WARN [ambari-client-thread-289] MITKerberosOperationHandler:459 - Failed to execute kadmin:
        Command: [/usr/bin/kadmin, -s, hadoop1.abc.com, -p, admin, -r, abc.com, -q, get_principal admin]
        ExitCode: 1
        STDOUT: Authenticating as principal admin with password.
        STDERR: kadmin: Client not found in Kerberos database while initializing kadmin interface
23 Dec 2016 15:17:30,452  INFO [ambari-client-thread-289] AbstractResourceProvider:810 - Caught an exception while updating host components, retrying : java.lang.IllegalArgumentException: Invalid KDC administrator credentials.
The KDC administrator credentials must be set as a persisted or temporary credential resource.This may be done by issuing a POST (or PUT for updating) to the /api/v1/clusters/:clusterName/credentials/kdc.admin.credential API entry point with the following payload:
{
  "Credential" : {
    "principal" : "(PRINCIPAL)", "key" : "(PASSWORD)", "type" : "(persisted|temporary)"}
  }
}
23 Dec 2016 15:17:30,703  INFO [ambari-client-thread-289] AbstractResourceProvider:925 - Received a updateHostComponent request, clusterName=FDOT_Hadoop, serviceName=RANGER_KMS, componentName=RANGER_KMS_SERVER, hostname=hadoop1.abc.com, request={ clusterName=FDOT_Hadoop, serviceName=RANGER_KMS, componentName=RANGER_KMS_SERVER, hostname=hadoop1.abc.com, desiredState=INSTALLED, state=null, desiredStackId=null, staleConfig=null, adminState=null}
^C
[root@hadoop1 ambari-server]# ^C
[root@hadoop1 ambari-server]#

[#2]

I have already tried this

-----
--- To set up Ambari's credential store, the following command must be invoked from the Ambari server host's command line:
--------------------------------------------------------------------------------------------------------------------------
[root@hadoop1 ambari-server]# ambari-server setup-security
Using python  /usr/bin/python
Security setup options...
===========================================================================
Choose one of the following options:
  [1] Enable HTTPS for Ambari server.
  [2] Encrypt passwords stored in ambari.properties file.
  [3] Setup Ambari kerberos JAAS configuration.
  [4] Setup truststore.
  [5] Import certificate to truststore.
===========================================================================
Enter choice, (1-5): 2
Please provide master key for locking the credential store:
Re-enter master key:
Do you want to persist master key. If you choose not to persist, you need to provide the Master Key while starting the ambari server as an env variable named AMBARI_SECURITY_MASTER_KEY or the start will prompt for the master key. Persist [y/n] (y)? y
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup-security' completed successfully.
[root@hadoop1 ambari-server]# ls -ltr /var/lib/ambari-server/keys/credentials.jceks
-rw-r----- 1 root root 503 Dec 23 15:33 /var/lib/ambari-server/keys/credentials.jceks
[root@hadoop1 ambari-server]#

---- TO TEST THE KEY STORED 
---------------------------
[root@hadoop1 ambari-server]# $JAVA_HOME/bin/keytool -list -keystore /var/lib/ambari-server/keys/credentials.jceks -storetype JCEKS
Enter keystore password:
Keystore type: JCEKS
Keystore provider: SunJCE
Your keystore contains 1 entry
ambari.db.password, Dec 23, 2016, SecretKeyEntry,



[root@hadoop1 ambari-server]#
[root@hadoop1 ambari-server]# $JAVA_HOME/bin/keytool -importpass \
 -keystore /var/lib/ambari-server/keys/credentials.jceks \
 -storetype JCEKS \
 -alias cluster.FDOT_hadoop.kdc.admin.credential
Enter keystore password:
Enter the password to be stored:
Re-enter password:
Enter key password for <cluster.FDOT_hadoop.kdc.admin.credential>
        (RETURN if same as keystore password):

[#3]

Hi @Sami Ahmad

Did you restart ambari server after doing security setup?

Thanks,

Deepak

[#4]

yes i did

[#5 · Accepted solution]

java.lang.IllegalArgumentException:Invalid KDC administrator credentials.

It appears that your KDC administrator credentials are incorrect. Please check with the KDC administrator for the correct principal name and password to use.

This issue is not related to the persisted credential store, which can be set up using option #2 in the ambari-server setup-security facility.

[#6]

but which credential we are talking about ? this error is coming up when I try to install ranger KMS

also how can I know what is my current KDC administrator credentials ?

[#7]

I reset the KDC credentials via the "Manage KDC credentials" button in Kerberos menu and now Iam getting a slightly different error when I try to reinstall Ranger KMS

my TGT system is working fine for HIVE n HBASE so why ranger KMS cant find the krb5.conf file . .is there a setting in the KMS service for this that might be wrong ?

        ... 103 more
23 Dec 2016 22:16:33,131  WARN [ambari-client-thread-837] ServletHandler:561 - Error Processing URI: /api/v1/clusters/FDOT_Hadoop/hosts/hadoop1.abc.com/host_components/RANGER_KMS_SERVER - (java.lang.RuntimeException) Update Host request submission failed: org.apache.ambari.server.AmbariException: The 'krb5-conf' configuration is not available
23 Dec 2016 22:16:33,131  WARN [ambari-client-thread-837] ServletHandler:561 - Error Processing URI: /api/v1/clusters/FDOT_Hadoop/hosts/hadoop1.abc.com/host_components/RANGER_KMS_SERVER - (java.lang.RuntimeException) Update Host request submission failed: org.apache.ambari.server.AmbariException: The 'krb5-conf' configuration is not available