AD Kerberos setup is failing in Create Keytabs step. Error Message: 2017-06-15 20:09:24,381 - Failed to create keytab for hive_llap-061517@TESTING.COM, missing cached file

[#1]

We have configured Hive LLAP in our existing environment for testing purpose. Cluster setup has been done using Service User Accounts present in AD. We tried to Enable Kerberos today and it got failed in Create Keytabs steps with below error message:

stderr: errors-257.txt

2017-06-15 20:09:24,381 - Failed to create keytab for hive_llap-061517@TESTING.COM, missing cached file

stdout: output-257.txt

2017-06-15 20:09:24,315 - Processing identities...
2017-06-15 20:09:24,378 - Creating keytab file for hive_llap-061517@TESTING.COM on host master1.TESTING.com
2017-06-15 20:09:24,380 - Creating keytab file for hive_llap-061517@TESTING.COM on host data1.TESTING.com

It seems that ambari is trying to create keytabs on linux hosts. we configured hive LLAP through ambari only.

Any help on this issue?

[#2 · Accepted solution]

It appears that there may be an _accounting_ issue in the Ambari database. This may have been created due to a previous failure when enabling Kerberos.

Try to remove the entries in the kerberos_principal_host and kerberos_principal where the principal_name column is hive_llap-061517@TESTING.COM in the Ambari database. Then restart Ambari and try to enabled Kerberos again.

[#3]

@Robert Levas

Thanks for the quick response. After removing entries in the kerberos_principal_host and kerberos_principal, it worked.

[#4]

@Shyam Shaw Awesome. I am glad it worked.

[#5]

1. Login to ambari database hosted server.

  2. Take the backup of database. Replace XXXXXX with correct pasword

nohup mysqldump -u root -pXXXXXX --databases ambari >/ambari.sql &

3. Login to mysql with root or ambari account and remove hive keytabs.

delete from kerberos_principal_host where principal_name like '%hive%';

delete from kerberos_principal where principal_name like '%hive%';

4. Restart Ambari server.

5. Regenerate the keytabs with valid   account

6. Start the Node manager.

Note: its not only for Hive.. we can remove based on error. as caches in ambari database prevents to regenerate again 

[#6]

Assuming all the other services are working properly except for Hive LLAP. You can try to 'Regenerate Keytabs' in Ambari. While doing that, you need to check 'Only regenerate keytabs for missing hosts and components' option.

If that doesn't work, you can regenerate all the keytabs for all hosts which requires all components to be restarted.

Also, if you are using a test AD KDC, you can try to restart it by following these instructions. Hope that helps.

[#7]

Hi @Robert Levas @Shyam Shaw, can you please help me with how to "remove the entries in the kerberos_principal_host and kerberos_principal where the principal_name column". Sorry for re-opening this, but I am facing a similar problem and unable to delete this from ambari database.

[#8]

@Rohan Goel

If you are using Ambari before version 2.7.0, then you can do the following:

delete from kerberos_principal_host where principal_name = 'THE PRINCIPAL NAME";
delete from kerberos_principal where principal_name = 'THE PRINCIPAL NAME";

If you are using Ambari, version 2.7.0 and above, then you have to do the following:

delete from kerberos_keytab_principal where principal_name = 'THE PRINCIPAL NAME";
delete from kerberos_principal where principal_name = 'THE PRINCIPAL NAME";

Then restart Ambari.

[#9]

Hello, 

I'm rencontring the same issue , I tried to remove keberos entries,but I still have the same issue. Any help please? 

2020-10-15T10:42:37.386+0200, User(admin), Operation(CUSTOM_COMMAND KERBEROS_CLIENT), Details(CHECK_KEYTABS KERBEROS/KERBEROS_CLIENT), Status(COMPLETED), RequestId(685), TaskId(7814), Hostname(ppla0016.angers.cnp.fr)
2020-10-15T10:42:37.421+0200, User(admin), Operation(CUSTOM_COMMAND KERBEROS_CLIENT), Details(CHECK_KEYTABS KERBEROS/KERBEROS_CLIENT), Status(COMPLETED), RequestId(685), TaskId(7810), Hostname(ppla0010.angers.cnp.fr)
2020-10-15T10:42:37.424+0200, User(admin), Operation(CUSTOM_COMMAND KERBEROS_CLIENT), Details(CHECK_KEYTABS KERBEROS/KERBEROS_CLIENT), Status(COMPLETED), RequestId(685), TaskId(7812), Hostname(ppla0012.angers.cnp.fr)
2020-10-15T10:42:37.955+0200, User(admin), Operation(EXECUTE AMBARI_SERVER_ACTION), Details(Create Keytabs), Status(QUEUED), RequestId(685), TaskId(7817), Hostname(null)
2020-10-15T10:42:37.964+0200, User(admin), Operation(EXECUTE AMBARI_SERVER_ACTION), Details(Create Keytabs), Status(IN_PROGRESS), RequestId(685), TaskId(7817), Hostname(null)
2020-10-15T10:42:39.552+0200, Operation(Keytab file creation), Status(Failed), Reason of failure(Failed to create keytab for clhdpi01-101520@P.AD.CNP.FR, missing cached file), RequestId(685), TaskId(7817), Principal(clhdpi01-101520@P.AD.CNP.FR), Hostname(ppla0017.angers.cnp.fr), Keytab file(/etc/security/keytabs/kerberos.service_check.101520.keytab)
2020-10-15T10:42:39.557+0200, User(admin), Operation(Kerberos Service Check), Status(FAILED), RequestId(685)
2020-10-15T10:42:39.557+0200, User(admin), Operation(EXECUTE AMBARI_SERVER_ACTION), Details(Create Keytabs), Status(FAILED), RequestId(685), TaskId(7817), Hostname(null)
2020-10-15T10:51:20.156+0200, User(admin), RemoteIp(10.68.123.240), Operation(Configuration change), RequestType(PUT), url(XXXXXXXX), ResultStatus(200 OK), VersionNumber(Vnull), VersionNote(null)
2020-10-15T10:51:20.519+0200, User(admin), RemoteIp(10.68.123.240), Operation(Service deletion), RequestType(DELETE), url(XXXXX), ResultStatus(200 OK), Service(KERBEROS)