Kerberized HDP 2.5 kerberos Service Check fails with error - Failed to create keytab for principal, missing cached file

[#1]

Hi,

In a kerberized HDP 2.5 cluster, when I try to run Service check from Ambari, getting the below mentioned error, always for ambari server host(If I try to kinit from putty, it works fine):

13 Feb 2017 19:29:46,088 INFO [ambari-client-thread-231] AmbariManagementControllerImpl:3749 - Received action execution request, clusterName=abc-123, request=isCommand :true, action :null, command :KERBEROS_SERVICE_CHECK, inputs :{}, resourceFilters: [RequestResourceFilter{serviceName='KERBEROS', componentName='null', hostNames=[]}], exclusive: false, clusterName :abc-123

13 Feb 2017 19:29:47,803 INFO [Server Action Executor Worker 4946] KerberosServerAction:352 - Processing identities...

13 Feb 2017 19:29:47,911 INFO [Server Action Executor Worker 4946] KerberosServerAction:456 - Processing identities completed.

13 Feb 2017 19:29:48,963 INFO [Server Action Executor Worker 4947] KerberosServerAction:352 - Processing identities...

13 Feb 2017 19:29:49,036 INFO [Server Action Executor Worker 4947] CreateKeytabFilesServerAction:193 - Creating keytab file for abc-123-021317@REALMNAME.COM on host abc-123-wn004.nix.REALMNAME.COM

13 Feb 2017 19:29:49,037 INFO [Server Action Executor Worker 4947] CreateKeytabFilesServerAction:193 - Creating keytab file for abc-123-021317@REALMNAME.COM on host abc-123-hn01.nix.REALMNAME.COM

13 Feb 2017 19:29:49,038 INFO [Server Action Executor Worker 4947] CreateKeytabFilesServerAction:193 - Creating keytab file for abc-123-021317@REALMNAME.COM on host abc-123-wn002.nix.REALMNAME.COM

13 Feb 2017 19:29:49,049 INFO [Server Action Executor Worker 4947] CreateKeytabFilesServerAction:193 - Creating keytab file for abc-123-021317@REALMNAME.COM on host abc-123-wn006.nix.REALMNAME.COM

13 Feb 2017 19:29:49,049 INFO [Server Action Executor Worker 4947] CreateKeytabFilesServerAction:193 - Creating keytab file for abc-123-021317@REALMNAME.COM on host abc-123-wn003.nix.REALMNAME.COM

13 Feb 2017 19:29:49,050 INFO [Server Action Executor Worker 4947] CreateKeytabFilesServerAction:193 - Creating keytab file for abc-123-021317@REALMNAME.COM on host abc-123-mn01.nix.REALMNAME.COM

13 Feb 2017 19:29:49,051 ERROR [Server Action Executor Worker 4947] CreateKeytabFilesServerAction:233 - Failed to create keytab for abc-123-021317@REALMNAME.COM, missing cached file

13 Feb 2017 19:29:49,052 INFO [Server Action Executor Worker 4947] KerberosServerAction:456 - Processing identities completed.

13 Feb 2017 19:29:49,993 ERROR [ambari-action-scheduler] ActionScheduler:428 - Operation completely failed, aborting request id: 216

I tried to disable and enable Kerberos, but it still fails.

[#2]

@Saurabh Singh

Did you already enabled Ambari-server for kerberos?

http://dev.hortonworks.com.s3.amazonaws.com/HDPDocuments/HDP2/HDP-2.5.0/bk_security/content/_set_up_...

In Ambari 2.4 - server by defaults creates and configure the principal for Ambari-server

Can you check kerberos_principal table and see whether entry present for that principal?

[#3]

No, I haven't set up kerberos specially for ambari server. I enabled Kerberos from Ambari and I am using an existing AD server.

This service check works on other nodes and it says something about missing cache.

[#4 · Accepted solution]

Apparently, as i mentioned above it was the cache thing. Today, on a new date, it tried again to create keytabs and it worked fine, since it wasn't looking into the cache. So my assumption is that somehow the cache got deleted yesterday, due to which it was failing.