Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Does cloudera automatically regenerate keytab files

Does cloudera automatically regenerate keytab files

Explorer


We had the following issues with statestored/catalogd:

F0306 00:42:57.593587 25457 init.cc:195] Failed to obtain Kerberos ticket for principal: impala/node-address@REALM.COM. Shell cmd: 'kinit -k -t /var/run/cloudera-scm-agent/process/2941-impala-STATESTORE/impala.keytab impala/node-address@REALM.COM 2>&1' exited with error status: '1'. Stdout was: 'kinit: Password incorrect while getting initial credentials
'
. Impalad exiting.


getprinc impala/node-address@REALM.COM listed that the principal has been modified shortly before the issue surface
And I saw that cloudera (the admin principal cloudera uses) has regenerated the keytab file


So what seems to have happened that it was done while Impala was running.
To fix the issue I stopped Impala -> Click regenerate kerberos credentials for statestored/catalogd -> Start Impala

That fixed the issue.

So my question is could this happen automatically. Does cloudera or impala regenerates kerberos credentials every half an year for example?Any suggestion what could be the reason. Or how to find out who/what trigger this?

 

I checked Recent commands tab in Cloudera but there were not commands at the time when kadmin logs reported that credential have been re-generated.

 

For reference kadmin.log from the time of the un-expected credentials regeneration: 

 

Mar 02 09:17:54 kdc-node kadmind[18275](Notice): Request: kadm5_init, cloudera-admin/admin@REALM.COM, success, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9, vers=3, flavor=6
Mar 02 09:17:54 kdc-node kadmind[18275](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9
Mar 02 09:17:54 kdc-node kadmind[18275](Notice): Request: kadm5_create_principal, impala/node-address.REALM.COM@REALM.COM, success, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9
Mar 02 09:17:54 kdc-node kadmind[18275](info): closing down fd 28
Mar 02 09:17:54 kdc-node kadmind[18275](Notice): Request: kadm5_init, cloudera-admin/admin@REALM.COM, success, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9, vers=3, flavor=6
Mar 02 09:17:54 kdc-node kadmind[18275](Notice): Request: kadm5_get_principal, impala/node-address.REALM.COM@REALM.COM, success, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9
Mar 02 09:17:54 kdc-node kadmind[18275](info): closing down fd 28
Mar 02 09:17:54 kdc-node kadmind[18275](Notice): Request: kadm5_init, cloudera-admin/admin@REALM.COM, success, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9, vers=3, flavor=6
Mar 02 09:17:54 kdc-node kadmind[18275](Notice): Request: kadm5_randkey_principal, impala/node-address.REALM.COM@REALM.COM, success, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9
Mar 02 09:17:54 kdc-node kadmind[18275](Notice): Request: kadm5_get_principal, impala/node-address.REALM.COM@REALM.COM, success, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9
Mar 02 09:17:54 kdc-node kadmind[18275](info): closing down fd 28
Mar 02 09:18:34 kdc-node kadmind[18275](Notice): Request: kadm5_init, cloudera-admin/admin@REALM.COM, success, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9, vers=3, flavor=6
Mar 02 09:18:34 kdc-node kadmind[18275](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9
Mar 02 09:18:34 kdc-node kadmind[18275](Notice): Request: kadm5_create_principal, impala/node-address@REALM.COM, Principal or policy already exists, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9
Mar 02 09:18:34 kdc-node kadmind[18275](info): closing down fd 28
Mar 02 09:18:34 kdc-node kadmind[18275](Notice): Request: kadm5_init, cloudera-admin/admin@REALM.COM, success, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9, vers=3, flavor=6
Mar 02 09:18:34 kdc-node kadmind[18275](Notice): Request: kadm5_get_principal, impala/node-address@REALM.COM, success, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9
Mar 02 09:18:34 kdc-node kadmind[18275](info): closing down fd 28
Mar 02 09:18:34 kdc-node kadmind[18275](Notice): Request: kadm5_init, cloudera-admin/admin@REALM.COM, success, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9, vers=3, flavor=6
Mar 02 09:18:34 kdc-node kadmind[18275](Notice): Request: kadm5_randkey_principal, impala/node-address@REALM.COM, success, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9
Mar 02 09:18:34 kdc-node kadmind[18275](Notice): Request: kadm5_get_principal, impala/node-address@REALM.COM, success, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9

2 REPLIES 2

Re: Does cloudera automatically regenerate keytab files

Explorer

Any ideas? 
In short:

Impala failed to authenticate against kerberos. 

It was due to keytab file having being regenerated somehow but not updated and Impala was using the old ones.

It didn't seem to have been done manually

To fix the issue I stopped Impala -> Click regenerate kerberos credentials for statestored/catalogd -> Start Impala

So my question is could this happen automatically ? How?

Re: Does cloudera automatically regenerate keytab files

Cloudera Employee

Hi Antoni,

 

 

No, this not supposed to happen automatically. Only CM can re-generate the credentials, and only on user request.

 

 

cheers,

zegab

Don't have an account?
Coming from Hortonworks? Activate your account here