I am using Ambari 2.0.1 and MIT Kerberos.
After running through the enabling Kerberos wizard, the services are failing to start. After some search I found out that there are no principals being created in the KDC:
"listprincs" just shows the previously (manually) created admin/admin@REALM principal, but no further principals as expected from enabling Kerberos via the wizard?!?!?!
This is the first time I see this strange behaviour, several other kerberized clusters didn't have this problem.
Why doesn't the Ambari wizard create principals in the KDC, while showing no errors at running through the wizard ?
Thanks in advance...
just wanted to drop you the note that I now have a Kerberos enabled cluster.
I just ignored the failure messages during service startup and wanted to deep dive into what is going on while Ambari creates principals and keytabs. I left the cluster in the stopped state including all errors ( ~60 red alerts)
To start the journey I ran in Ambari=>Admin=>Kerberos "regenerate keytabs"
Surprisingly this triggered the creation of principals and keytabs successfully and I ended up in the state I expected from the Wizard, to have all the required principals and keytabs on the corresponding hosts.
Anyway, after the "regenerate keytabs" I was able to successfully start all the services.
Hello @mahadev ,
no, the logs aren't that verbose, I just see:
12 Dec 2015 20:38:51,803 ERROR [ambari-action-scheduler] ClusterImpl:2382 - ServiceComponentHost lookup exception 12 Dec 2015 20:38:51,810 INFO [Server Action Executor Worker 1179] KerberosServerAction:327 - Processing identities... 12 Dec 2015 20:38:52,032 INFO [Server Action Executor Worker 1179] KerberosServerAction:429 - Processing identities completed. 12 Dec 2015 20:38:52,839 ERROR [ambari-action-scheduler] ClusterImpl:2382 - ServiceComponentHost lookup exception 12 Dec 2015 20:38:52,847 INFO [Server Action Executor Worker 1180] KerberosServerAction:327 - Processing identities... 12 Dec 2015 20:38:52,848 INFO [Server Action Executor Worker 1180] CreateKeytabFilesServerAction:170 - Creating keytab file for HTTP/deala01875.domain@HDP.SIT on host deala01875.comain
And afterwards the keytab files are being created...
I ran a "disable Kerberos" afterwards, and then in the ambari-logs I can see that it tries to delete all the principals, but somehow the creation fails. I tried the whole steps several times....
Is Ambari running as non-root? If so, make sure the user that Ambari runs as, has authorization to write to /var/lib/ambari-server/tmp.
Also, did you change the value of the encryption types field? I think that it is under Advanced Kerberos-env section when on the Configure Kerberos page of the wizard. If this field is blank, then no encryption types will be specified and thus no keytab entries for the keytab files will be generated. If you edited the field, try setting it back to aes des3-cbc-sha1 rc4 des-cbc-md5 when you enable Kerberos again.
Hi Robert, thanks for answering.
Ambari was running as root. "was" because I did a reinstall from scratch in the meantime due to time pressure for delivering the cluster....unfortunately.
This new installation worked nice, therefore I guess the problem was caused by "disable Kerberos" and maybe that was done in a corrupt manner...?!?!
Hello, I am again facing the same issue while enabling Kerberos on a newly installed cluster => no principals are being created and no keytabs are generated, although the enable Kerberos wizard tells so ?!?!?
I didn't edit the encryption type field in the Kerberos-wizard and Ambari is running as root, therefore it will be able to write to /var/lib/ambari-server/tmp
Ambari logfile states the creation of the keytab files:
25 Jan 2016 12:01:48,279 INFO [Server Action Executor Worker 2148] CreateKeytabFilesServerAction:170 - Creating keytab file for HTTP/b0d05g22.<domain>@<realm> on host b0d05g22.<domain> 25 Jan 2016 12:01:48,280 INFO [Server Action Executor Worker 2148] CreateKeytabFilesServerAction:170 - Creating keytab file for hdfs@<realm> on host b0d05g22.<domain>
but at the end, no keytab file is being deployed, and also no principal has been created.
If I check the principals AFTER the Kerberos-Wizard has "successfully" created them, none of the are in the KDC =>
sudo kadmin.localkadmin.local: listprincs K/M@<realm> admin/admin@<realm> kadmin/admin@<realm> kadmin/b0d095j2.<domain>@<realm> kadmin/changepw@<realm> krbtgt/HDP.ZURICH.PRD@<realm> kadmin.local:
Kerberos client conf contains:
[libdefaults] renew_lifetime = 7d forwardable = true default_realm = <realm> ticket_lifetime = 24h dns_lookup_realm = false dns_lookup_kdc = false #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5 #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
What to check further...Any hint highly appreciated...
To remove the Kerberos Descriptor:
curl -u admin:admin -i -H 'X-Requested-By: ambari' -X DELETE http://AMBARI_SERVER:8080/api/v1/clusters/CLUSTER_NAME/artifacts/kerberos_descriptor
Hi @Robert Levas , thanks for this hint.
I did exactly that, but ended up in the same situation.
No principals have been created and no keytabs have been deployed, although the wizard marked every step as "green" until starting up the services