Support Questions

Find answers, ask questions, and share your expertise

Ambari Fails to create Keytabs when Installing new services or when trying to regenerate keytabs of existing services

avatar
Explorer

I am getting a server action failed error when ambari tries to create keytabs. I have no idea what is causing this error. I am using KDC admin to install the service. Please help. @Geoffrey Shelton Okot

96567-keytab-error.png

1 ACCEPTED SOLUTION

avatar
Master Mentor

@huzaira bashir

There seems to be a mismatch between the Active Directory encryption type and the MIT encryption types can you align the 2 supported_enctypes to be the same.

Windows supports the below encryption types depending on the Windows version which are weak encryption

DES_CBC_CRC 
DES_CBC_MD5 
RC4_HMAC_MD5 
AES128_HMAC_SHA1 
AES256_HMAC_SHA1 

In your kdc.conf you have stronger encryption types you validate the AD encryption types? Else try to comment out the kdc.conf encryptions see below

# supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal camellia256-cts:normal camellia128-cts:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal

and see whether the error persists?

View solution in original post

27 REPLIES 27

avatar
Explorer

@Geoffrey Shelton Okot @Robert Levas

Attached is the error in the ambari-server log when trying to regenerate keytab for hbase

97532-error-in-ambari-server-log.png

avatar
Master Mentor

@huzaira bashir

Can you share the corresponding to my screenshots photo5 and 6 , I built a VM to test your case and documented all the steps over the weekend and I am surprised it can't work for you.
Is Java Cryptography Extension (JCE) installed, check the syntax below please adjust your jdk_home accordingly

# zipgrep CryptoAllPermission /usr/jdk64/jdk1.8.0_112/jre/lib/security/local_policy.jar

The desired output should be

default_local.policy: permission javax.crypto.CryptoAllPermission;

avatar

@huzaira bashir

It seems like there may be an issue exporting keytab files from the KDC. Can you try to do this manually using the admin user you configured in Ambari?

/bin/kinit -c /tmp/my_cc -S kadmin/<KDC Admin Server Host> <KDC Admin Principal Name>

/bin/kadmin -c /tmp/my_cc -s <KDC Admin Server Host> -r <Realm> -q "xst -k /tmp/ambari_tmp.keytab -e des3-cbc-sha1-kd:normal,rc4-hmac:normal,des-cbc-md5:normal,aes128-cts-hmac-sha1-96:normal,aes256-cts-hmac-sha1-96:normal ambari-qa-<Cluster Name>@<Realm>"

/bin/klist -kte /tmp/ambari_tmp.keytab

You will need to change the specifics to match your cluster. For my cluster I am using:

  • KDC Admin Server Host: c7402.ambari.apache.org
  • KDC Admin Principal Name: admin/admin!@EXAMPLE.COM
  • Realm: EXAMPLE.COM
  • Cluster Name: c1

Also, I assume that you haven't changed:

  • The default temporary directory: /tmp
  • The kadmin principal name: kadmin/<KDC Admin Server Host>
  • The format of the Ambari smoke user principal name: ambari-qa-<Cluster Name>@<Realm>
  • The default encryption types: aes des3-cbc-sha1 rc4 des-cbc-md5

Using the commands from above, I get the following:

[root@c7402 ~]# /bin/kinit -c /tmp/my_cc -S kadmin/c7402.ambari.apache.org admin/admin@EXAMPLE.COM
Password for admin/admin@EXAMPLE.COM:<br>[root@c7402 ~]# /bin/kadmin -c /tmp/my_cc -s c7402.ambari.apache.org -r EXAMPLE.COM -q "xst -k "/tmp/ambari_tmp.keytab" -e des3-cbc-sha1-kd:normal,rc4-hmac:normal,des-cbc-md5:normal,aes128-cts-hmac-sha1-96:normal,aes256-cts-hmac-sha1-96:normal ambari-server-c1@EXAMPLE.COM"
Authenticating as principal admin/admin@EXAMPLE.COM with existing credentials.
Entry for principal ambari-server-c1@EXAMPLE.COM with kvno 4, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/ambari_tmp.keytab.
Entry for principal ambari-server-c1@EXAMPLE.COM with kvno 4, encryption type arcfour-hmac added to keytab WRFILE:/tmp/ambari_tmp.keytab.
Entry for principal ambari-server-c1@EXAMPLE.COM with kvno 4, encryption type des-cbc-md5 added to keytab WRFILE:/tmp/ambari_tmp.keytab.
Entry for principal ambari-server-c1@EXAMPLE.COM with kvno 4, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/ambari_tmp.keytab.
Entry for principal ambari-server-c1@EXAMPLE.COM with kvno 4, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/ambari_tmp.keytab.
Administration credentials NOT DESTROYED.<br>[root@c7402 ~]# /bin/klist -kte /tmp/ambari_tmp.keytab
Keytab name: FILE:/tmp/ambari_tmp.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   4 01/09/2019 15:31:29 ambari-server-c1@EXAMPLE.COM (des3-cbc-sha1)
   4 01/09/2019 15:31:29 ambari-server-c1@EXAMPLE.COM (arcfour-hmac)
   4 01/09/2019 15:31:29 ambari-server-c1@EXAMPLE.COM (des-cbc-md5)
   4 01/09/2019 15:31:29 ambari-server-c1@EXAMPLE.COM (aes128-cts-hmac-sha1-96)
   4 01/09/2019 15:31:29 ambari-server-c1@EXAMPLE.COM (aes256-cts-hmac-sha1-96)

Can you try this to see if you get any errors?

avatar
Explorer

Hi @Robert Levas,

I am able to do this manually. I got no errors.

97573-manual-process.png

avatar
Explorer

@Geoffrey Shelton Okot

Hi, my Javax policy is fine. see attached. Also I dont see photo 5 and 6 that you mentioned in your previous comment.

97559-javax-policy.png

avatar
Master Mentor

@huzaira bashir

Here they are


photo05.pngphoto06.png

avatar
Contributor

@Geoffrey Shelton Okot @huzaira bashir

Did you manage to solve this yet?

What was the problem?

avatar
Master Mentor

@Javert Kirilov

Have you encountered a Kerberos issue? Is it similar to the thread opened by Bashir? It's usually good to open a new thread as this one isn't being actively viewed.
I will be happy to help you if you attached some logs ,screenshot and some error logs

avatar
Contributor

Thank you, I had this exact issue with same errors and nothing in the comment discussion helped. However, after several ambari-server restarts and dumb retries of the "Kerberos wizard" with similar settings magically resolved this. I'm not sure at all what was the problem..

avatar
Explorer

@Robert Levas @Geoffrey Shelton Okot when I used debug mode I get an additional line which might point to an encryption issue? please see attached pic

97607-debug-error-message.png