@David Tam
I think I found the issue after walking through the steps in the doc you provided - https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_command-line-installation/content/config....
The documentation appears to be incorrect and there is an inconsistency with the Livy Kerberos identity.
Step 8a contains
kadmin.local -q "addprinc -randkey livy@EXAMPLE.COM"
kadmin.local -q "xst -k /etc/security/keytabs/livy.headless.keytab livy@EXAMPLE.COM"
Then step 8c contains
livy.server.launch.kerberos.keytab /etc/security/keytabs/livy.headless.keytab
livy.server.launch.kerberos.principal livy/_HOST@EXAMPLE.COM
The disconnect is the principal name: livy@EXAMPLE.COM vs livy/_HOST@EXAMPLE.COM. One or the other needs to be used.
Technically, Livy Server is not an interactive user, so a service principal should be created for it. So I would choose to use the livy/_HOST@EXAMPLE.COM representation of the Kerberos identity. However it might be easier to choose the user identity and go with livy@EXAMPLE.COM. In any case, the data in the configuration needs to be consistent.
If you chose to go with the user Kerberos identity, then the following values need to be set in the livy.conf file:
livy.server.launch.kerberos.keytab /etc/security/keytabs/livy.headless.keytab
livy.server.launch.kerberos.principal livy@EXAMPLE.COM
I assume that the user principal has already been created in the KDC and the keytab file was created using it. So after restarting the Livy Server, all should work.
If you choose to go the service principal route, then the following values need to be set in the livy.conf file:
livy.server.launch.kerberos.keytab /etc/security/keytabs/livy.service.keytab
livy.server.launch.kerberos.principal livy/_HOST@EXAMPLE.COM
However, I am not sure if Livy automatically translates _HOST to the relevant host's name. So it might be safer (but less portable) to explicitly put the host name there. You can get the correct hostname by issuing a the following command
hostname -f
For example:
[root@c6403 ~]# hostname -f
c6403.ambari.apache.org
Using this value, manually replace _HOST:
livy.server.launch.kerberos.keytab /etc/security/keytabs/livy.service.keytab
livy.server.launch.kerberos.principal livy/c6403.ambari.apache.org@EXAMPLE.COM
Then you need to create the relevant Kerberos principal and keytab file,. If possible, you can do this from the Livy server host, using kadmin, rather than kadmin.local:
kadmin -p <kdc admin principal> -q "addprinc -randkey livy/`hostname -f`@EXAMPLE.COM"
kadmin -p <kdc admin principal> -q "xst -k /etc/security/keytabs/livy.service.keytab livy/`hostname -f`@EXAMPLE.COM"
For example:
[root@c6403 ~]# kadmin -p admin/admin -q "addprinc -randkey livy/`hostname -f`@EXAMPLE.COM"
Authenticating as principal admin/admin with password.
Password for admin/admin@EXAMPLE.COM:
WARNING: no policy specified for livy/c6403.ambari.apache.org@EXAMPLE.COM; defaulting to no policy
Principal "livy/c6403.ambari.apache.org@EXAMPLE.COM" created.
[root@c6403 ~]# kadmin -p admin/admin -q "xst -k /etc/security/keytabs/livy.headless.keytab livy/`hostname -f`@EXAMPLE.COM"
Authenticating as principal admin/admin with password.
Password for admin/admin@EXAMPLE.COM:
Entry for principal livy/c6403.ambari.apache.org@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/security/keytabs/livy.headless.keytab.
Entry for principal livy/c6403.ambari.apache.org@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/security/keytabs/livy.headless.keytab.
Entry for principal livy/c6403.ambari.apache.org@EXAMPLE.COM with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/etc/security/keytabs/livy.headless.keytab.
Entry for principal livy/c6403.ambari.apache.org@EXAMPLE.COM with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/etc/security/keytabs/livy.headless.keytab.
Entry for principal livy/c6403.ambari.apache.org@EXAMPLE.COM with kvno 2, encryption type des-hmac-sha1 added to keytab WRFILE:/etc/security/keytabs/livy.headless.keytab.
Entry for principal livy/c6403.ambari.apache.org@EXAMPLE.COM with kvno 2, encryption type des-cbc-md5 added to keytab WRFILE:/etc/security/keytabs/livy.headless.keytab.
If you need to use kadmin.local on the KDC server host, you can do the following the manually copy the resulting keytab file into place on the Livy server host:
kadmin.local -q "addprinc -randkey livy/`hostname -f`@EXAMPLE.COM"
kadmin.local -q "xst -k /etc/security/keytabs/livy.service.keytab livy/`hostname -f`@EXAMPLE.COM"
Meanwhile I will see if the documentation can be fixed.
