We had the following issues with statestored/catalogd:
F0306 00:42:57.593587 25457 init.cc:195] Failed to obtain Kerberos ticket for principal: impala/node-address@REALM.COM. Shell cmd: 'kinit -k -t /var/run/cloudera-scm-agent/process/2941-impala-STATESTORE/impala.keytab impala/node-address@REALM.COM 2>&1' exited with error status: '1'. Stdout was: 'kinit: Password incorrect while getting initial credentials
'
. Impalad exiting.
getprinc impala/node-address@REALM.COM listed that the principal has been modified shortly before the issue surface
And I saw that cloudera (the admin principal cloudera uses) has regenerated the keytab file
So what seems to have happened that it was done while Impala was running.
To fix the issue I stopped Impala -> Click regenerate kerberos credentials for statestored/catalogd -> Start Impala
That fixed the issue.
So my question is could this happen automatically. Does cloudera or impala regenerates kerberos credentials every half an year for example?Any suggestion what could be the reason. Or how to find out who/what trigger this?
I checked Recent commands tab in Cloudera but there were not commands at the time when kadmin logs reported that credential have been re-generated.
For reference kadmin.log from the time of the un-expected credentials regeneration:
Mar 02 09:17:54 kdc-node kadmind[18275](Notice): Request: kadm5_init, cloudera-admin/admin@REALM.COM, success, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9, vers=3, flavor=6
Mar 02 09:17:54 kdc-node kadmind[18275](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9
Mar 02 09:17:54 kdc-node kadmind[18275](Notice): Request: kadm5_create_principal, impala/node-address.REALM.COM@REALM.COM, success, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9
Mar 02 09:17:54 kdc-node kadmind[18275](info): closing down fd 28
Mar 02 09:17:54 kdc-node kadmind[18275](Notice): Request: kadm5_init, cloudera-admin/admin@REALM.COM, success, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9, vers=3, flavor=6
Mar 02 09:17:54 kdc-node kadmind[18275](Notice): Request: kadm5_get_principal, impala/node-address.REALM.COM@REALM.COM, success, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9
Mar 02 09:17:54 kdc-node kadmind[18275](info): closing down fd 28
Mar 02 09:17:54 kdc-node kadmind[18275](Notice): Request: kadm5_init, cloudera-admin/admin@REALM.COM, success, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9, vers=3, flavor=6
Mar 02 09:17:54 kdc-node kadmind[18275](Notice): Request: kadm5_randkey_principal, impala/node-address.REALM.COM@REALM.COM, success, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9
Mar 02 09:17:54 kdc-node kadmind[18275](Notice): Request: kadm5_get_principal, impala/node-address.REALM.COM@REALM.COM, success, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9
Mar 02 09:17:54 kdc-node kadmind[18275](info): closing down fd 28
Mar 02 09:18:34 kdc-node kadmind[18275](Notice): Request: kadm5_init, cloudera-admin/admin@REALM.COM, success, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9, vers=3, flavor=6
Mar 02 09:18:34 kdc-node kadmind[18275](Notice): Request: kadm5_get_policy, default, Policy does not exist, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9
Mar 02 09:18:34 kdc-node kadmind[18275](Notice): Request: kadm5_create_principal, impala/node-address@REALM.COM, Principal or policy already exists, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9
Mar 02 09:18:34 kdc-node kadmind[18275](info): closing down fd 28
Mar 02 09:18:34 kdc-node kadmind[18275](Notice): Request: kadm5_init, cloudera-admin/admin@REALM.COM, success, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9, vers=3, flavor=6
Mar 02 09:18:34 kdc-node kadmind[18275](Notice): Request: kadm5_get_principal, impala/node-address@REALM.COM, success, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9
Mar 02 09:18:34 kdc-node kadmind[18275](info): closing down fd 28
Mar 02 09:18:34 kdc-node kadmind[18275](Notice): Request: kadm5_init, cloudera-admin/admin@REALM.COM, success, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9, vers=3, flavor=6
Mar 02 09:18:34 kdc-node kadmind[18275](Notice): Request: kadm5_randkey_principal, impala/node-address@REALM.COM, success, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9
Mar 02 09:18:34 kdc-node kadmind[18275](Notice): Request: kadm5_get_principal, impala/node-address@REALM.COM, success, client=cloudera-admin/admin@REALM.COM, service=kadmin/kdc-node@REALM.COM, addr=10.153.201.9
