Created on 02-13-2017 06:48 PM - edited 09-16-2022 04:04 AM
Hi,
In a kerberized HDP 2.5 cluster, when I try to run Service check from Ambari, getting the below mentioned error, always for ambari server host(If I try to kinit from putty, it works fine):
13 Feb 2017 19:29:46,088 INFO [ambari-client-thread-231] AmbariManagementControllerImpl:3749 - Received action execution request, clusterName=abc-123, request=isCommand :true, action :null, command :KERBEROS_SERVICE_CHECK, inputs :{}, resourceFilters: [RequestResourceFilter{serviceName='KERBEROS', componentName='null', hostNames=[]}], exclusive: false, clusterName :abc-123
13 Feb 2017 19:29:47,803 INFO [Server Action Executor Worker 4946] KerberosServerAction:352 - Processing identities...
13 Feb 2017 19:29:47,911 INFO [Server Action Executor Worker 4946] KerberosServerAction:456 - Processing identities completed.
13 Feb 2017 19:29:48,963 INFO [Server Action Executor Worker 4947] KerberosServerAction:352 - Processing identities...
13 Feb 2017 19:29:49,036 INFO [Server Action Executor Worker 4947] CreateKeytabFilesServerAction:193 - Creating keytab file for abc-123-021317@REALMNAME.COM on host abc-123-wn004.nix.REALMNAME.COM
13 Feb 2017 19:29:49,037 INFO [Server Action Executor Worker 4947] CreateKeytabFilesServerAction:193 - Creating keytab file for abc-123-021317@REALMNAME.COM on host abc-123-hn01.nix.REALMNAME.COM
13 Feb 2017 19:29:49,038 INFO [Server Action Executor Worker 4947] CreateKeytabFilesServerAction:193 - Creating keytab file for abc-123-021317@REALMNAME.COM on host abc-123-wn002.nix.REALMNAME.COM
13 Feb 2017 19:29:49,049 INFO [Server Action Executor Worker 4947] CreateKeytabFilesServerAction:193 - Creating keytab file for abc-123-021317@REALMNAME.COM on host abc-123-wn006.nix.REALMNAME.COM
13 Feb 2017 19:29:49,049 INFO [Server Action Executor Worker 4947] CreateKeytabFilesServerAction:193 - Creating keytab file for abc-123-021317@REALMNAME.COM on host abc-123-wn003.nix.REALMNAME.COM
13 Feb 2017 19:29:49,050 INFO [Server Action Executor Worker 4947] CreateKeytabFilesServerAction:193 - Creating keytab file for abc-123-021317@REALMNAME.COM on host abc-123-mn01.nix.REALMNAME.COM
13 Feb 2017 19:29:49,051 ERROR [Server Action Executor Worker 4947] CreateKeytabFilesServerAction:233 - Failed to create keytab for abc-123-021317@REALMNAME.COM, missing cached file
13 Feb 2017 19:29:49,052 INFO [Server Action Executor Worker 4947] KerberosServerAction:456 - Processing identities completed.
13 Feb 2017 19:29:49,993 ERROR [ambari-action-scheduler] ActionScheduler:428 - Operation completely failed, aborting request id: 216
I tried to disable and enable Kerberos, but it still fails.
Created 02-14-2017 01:55 PM
Apparently, as i mentioned above it was the cache thing. Today, on a new date, it tried again to create keytabs and it worked fine, since it wasn't looking into the cache. So my assumption is that somehow the cache got deleted yesterday, due to which it was failing.
Created 02-13-2017 07:10 PM
Did you already enabled Ambari-server for kerberos?
In Ambari 2.4 - server by defaults creates and configure the principal for Ambari-server
Can you check kerberos_principal table and see whether entry present for that principal?
Created 02-14-2017 01:44 PM
No, I haven't set up kerberos specially for ambari server. I enabled Kerberos from Ambari and I am using an existing AD server.
This service check works on other nodes and it says something about missing cache.
Created 02-14-2017 01:55 PM
Apparently, as i mentioned above it was the cache thing. Today, on a new date, it tried again to create keytabs and it worked fine, since it wasn't looking into the cache. So my assumption is that somehow the cache got deleted yesterday, due to which it was failing.