Created on 01-03-2019 02:21 PM - edited 09-16-2022 07:01 AM
I am getting a server action failed error when ambari tries to create keytabs. I have no idea what is causing this error. I am using KDC admin to install the service. Please help. @Geoffrey Shelton Okot
Created 01-16-2019 11:57 AM
There seems to be a mismatch between the Active Directory encryption type and the MIT encryption types can you align the 2 supported_enctypes to be the same.
Windows supports the below encryption types depending on the Windows version which are weak encryption
DES_CBC_CRC DES_CBC_MD5 RC4_HMAC_MD5 AES128_HMAC_SHA1 AES256_HMAC_SHA1
In your kdc.conf you have stronger encryption types you validate the AD encryption types? Else try to comment out the kdc.conf encryptions see below
# supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
and see whether the error persists?
Created 01-03-2019 02:57 PM
Can you take a look at your Ambari server log (/var/log/ambari-server/ambari-server.log) and see if there are any interesting error messages?
Created 01-03-2019 05:00 PM
What is the HDP version? The screenshot doesn't look a typical MIT Kerberos enabling UI? Could you be using the AD as KDC? Having said that can you share how you procedure used? Can you share the Kerberos enabling screenshots from Ambari?
If you could answer promptly with the above info then it would help a great deal.
HTH
Created on 01-04-2019 06:46 AM - edited 08-17-2019 03:12 PM
The screenshot shared previously was of Ambari UI when adding hbase as a service.
Kerberos is MIT and was working fine when initially enabled. I realized there was a problem when adding a new service to the cluster. Ambari creates the principles needed in the kerberos db but is unable to create the keytabs.
HDP version is 3.0.0.0-1634
Ambari Version 2.7.1.0
The procedure i used was the one in the guidelines of ambari using an exsisting MIT kerberos. It works fine until you try and add a new service or regenerate keytabs for an existing service.
@Robert Levas There are no error messages in the logs that say anything.
Created 01-04-2019 02:15 PM
At least I am reassured about the previous screenshot. From the screenshot, I don't see domain which should be in the format and comma separated if your REALM is TEST.COM note the dot(.)
.test.com,test.com
And the Kadmin too, meanwhile can you share a tokenized version of you krb5.conf,kdc.conf and kadm5.acl most important ensure these 2 daemons are running
Enable auto start
# systemctl enable krb5kdc # systemctl enable kadmin
Start the daemons
# /etc/rc.d/init.d/krb5kdc start # /etc/rc.d/init.d/kadmin start
or
# systemctl start krb5kdc # systemctl start kadmin
Whichever is applicable
HTH
Created 01-04-2019 02:35 PM
Is the Ambari server on a host that is registered with the cluster? If not, I was recently alerted to an issue where this case was causing an error. But enabling Kerberos would have failed for you... unless you had enabled Kerberos before upgrading to Ambari 2.7.1. See AMBARI-25088 - Enable Kerberos fails when Ambari server is not on a registered host.
Created 01-07-2019 08:43 AM
Hi @Robert Levas the server is on the same domain. Kerberos is ok. the only issue is generation of keytabs when I add a new service. The principals are also created it just fails at keytab generation.
Created 01-05-2019 10:39 PM
Created 01-05-2019 10:41 PM
@huzaira bashir
Please follow the steps and update this thread, I am sure there is a step you missed follow page by page. On your screenshot I didn't see the Domain
Created on 01-07-2019 08:41 AM - edited 08-17-2019 03:12 PM
Hi have done everything accordingly, the domain is defined in the krb5.conf file. The ACL file is also ok.
my process are running as expected. the only problem is Ambari is not able to generate keytabs. Like I said, Ambari creates the principals required in the kerberos db but is unable to generate keytabs..
I can run Kadmin and list the princs.
Created on 01-07-2019 08:52 AM - edited 08-17-2019 03:12 PM
@Geoffrey Shelton Okot @Robert Levas
Attached is the error in the ambari-server log when trying to regenerate keytab for hbase
Created 01-07-2019 09:35 AM
Can you share the corresponding to my screenshots photo5 and 6 , I built a VM to test your case and documented all the steps over the weekend and I am surprised it can't work for you.
Is Java Cryptography Extension (JCE) installed, check the syntax below please adjust your jdk_home accordingly
# zipgrep CryptoAllPermission /usr/jdk64/jdk1.8.0_112/jre/lib/security/local_policy.jar
The desired output should be
default_local.policy: permission javax.crypto.CryptoAllPermission;
Created 01-09-2019 03:34 PM
It seems like there may be an issue exporting keytab files from the KDC. Can you try to do this manually using the admin user you configured in Ambari?
/bin/kinit -c /tmp/my_cc -S kadmin/<KDC Admin Server Host> <KDC Admin Principal Name> /bin/kadmin -c /tmp/my_cc -s <KDC Admin Server Host> -r <Realm> -q "xst -k /tmp/ambari_tmp.keytab -e des3-cbc-sha1-kd:normal,rc4-hmac:normal,des-cbc-md5:normal,aes128-cts-hmac-sha1-96:normal,aes256-cts-hmac-sha1-96:normal ambari-qa-<Cluster Name>@<Realm>" /bin/klist -kte /tmp/ambari_tmp.keytab
You will need to change the specifics to match your cluster. For my cluster I am using:
Also, I assume that you haven't changed:
Using the commands from above, I get the following:
[root@c7402 ~]# /bin/kinit -c /tmp/my_cc -S kadmin/c7402.ambari.apache.org admin/admin@EXAMPLE.COM Password for admin/admin@EXAMPLE.COM:<br>[root@c7402 ~]# /bin/kadmin -c /tmp/my_cc -s c7402.ambari.apache.org -r EXAMPLE.COM -q "xst -k "/tmp/ambari_tmp.keytab" -e des3-cbc-sha1-kd:normal,rc4-hmac:normal,des-cbc-md5:normal,aes128-cts-hmac-sha1-96:normal,aes256-cts-hmac-sha1-96:normal ambari-server-c1@EXAMPLE.COM" Authenticating as principal admin/admin@EXAMPLE.COM with existing credentials. Entry for principal ambari-server-c1@EXAMPLE.COM with kvno 4, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/ambari_tmp.keytab. Entry for principal ambari-server-c1@EXAMPLE.COM with kvno 4, encryption type arcfour-hmac added to keytab WRFILE:/tmp/ambari_tmp.keytab. Entry for principal ambari-server-c1@EXAMPLE.COM with kvno 4, encryption type des-cbc-md5 added to keytab WRFILE:/tmp/ambari_tmp.keytab. Entry for principal ambari-server-c1@EXAMPLE.COM with kvno 4, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/ambari_tmp.keytab. Entry for principal ambari-server-c1@EXAMPLE.COM with kvno 4, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/ambari_tmp.keytab. Administration credentials NOT DESTROYED.<br>[root@c7402 ~]# /bin/klist -kte /tmp/ambari_tmp.keytab Keytab name: FILE:/tmp/ambari_tmp.keytab KVNO Timestamp Principal ---- ------------------- ------------------------------------------------------ 4 01/09/2019 15:31:29 ambari-server-c1@EXAMPLE.COM (des3-cbc-sha1) 4 01/09/2019 15:31:29 ambari-server-c1@EXAMPLE.COM (arcfour-hmac) 4 01/09/2019 15:31:29 ambari-server-c1@EXAMPLE.COM (des-cbc-md5) 4 01/09/2019 15:31:29 ambari-server-c1@EXAMPLE.COM (aes128-cts-hmac-sha1-96) 4 01/09/2019 15:31:29 ambari-server-c1@EXAMPLE.COM (aes256-cts-hmac-sha1-96)
Can you try this to see if you get any errors?
Created on 01-10-2019 07:13 AM - edited 08-17-2019 03:11 PM
Created on 01-09-2019 06:38 AM - edited 08-17-2019 03:12 PM
Hi, my Javax policy is fine. see attached. Also I dont see photo 5 and 6 that you mentioned in your previous comment.
Created 01-09-2019 08:48 AM
Created 01-15-2019 12:21 PM
Created 01-15-2019 03:51 PM
Have you encountered a Kerberos issue? Is it similar to the thread opened by Bashir? It's usually good to open a new thread as this one isn't being actively viewed.
I will be happy to help you if you attached some logs ,screenshot and some error logs
Created 01-15-2019 03:54 PM
Thank you, I had this exact issue with same errors and nothing in the comment discussion helped. However, after several ambari-server restarts and dumb retries of the "Kerberos wizard" with similar settings magically resolved this. I'm not sure at all what was the problem..
Created on 01-16-2019 08:05 AM - edited 08-17-2019 03:11 PM
@Robert Levas @Geoffrey Shelton Okot when I used debug mode I get an additional line which might point to an encryption issue? please see attached pic
Created 01-16-2019 01:25 PM
Interesting.... Actually I am not sure why `UNKNOWN:normal` is in there. Did you edit the encryption types in the configure Kerberos page, under Advanced kerberos-env? It should be `aes des3-cbc-sha1 rc4 des-cbc-md5`. Does your have a different value?