Support Questions

Find answers, ask questions, and share your expertise

kerberos livy :- "requirement failed: Kerberos requires livy.server.kerberos.keytab to be provided."

avatar
Super Collaborator

Hello I am using kerberoized HDP 2.5.3 and I am trying out livy.

I have setup according to https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_command-line-installation/content/config... and without kerberos livy server can startup and response to http request.

However if I add the kerberos section in livy.conf then I get this in the log:

[root@master conf]# tail -1000f /var/log/livy/livy-livy-server.out
log4j:WARN No appenders could be found for logger (com.cloudera.livy.server.LivyServer).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: Kerberos requires livy.server.kerberos.keytab to be provided.
        at scala.Predef$.require(Predef.scala:233)
        at com.cloudera.livy.server.LivyServer.runKinit(LivyServer.scala:173)
        at com.cloudera.livy.server.LivyServer.start(LivyServer.scala:134)
        at com.cloudera.livy.server.LivyServer$.main(LivyServer.scala:277)
        at com.cloudera.livy.server.LivyServer.main(LivyServer.scala)


Any idea? Following is the livy.conf

[root@master conf]# cat /etc/livy/conf/livy.conf
livy.spark.master = yarn
livy.spark.deployMode = cluster
livy.environment production
livy.impersonation.enabled true
livy.server.csrf_protection.enabled true
livy.server.port 8998
livy.server.session.timeout 3600000
livy.server.recovery.mode off


#livy.server.auth.type = kerberos
livy.server.launch.kerberos.keytab = /etc/security/keytabs/livy.headless.keytab
livy.server.launch.kerberos.principal = livy/_HOST@LBG.COM


#livy.server.kerberos.keytab = /etc/security/keytabs/livy.headless.keytab
livy.server.auth.kerberos.keytab /etc/security/keytabs/spnego.service.keytab
livy.server.auth.kerberos.principal HTTP/_HOST@LBG.COM


livy.superusers=livy


Thank you in advance!

7 REPLIES 7

avatar

@David Tam

Can you verify that /etc/security/keytabs/livy.headless.keytab exists on the host where Livy is installed? Also make sure that the file is readable by the user executing the Livy Server.

If the file does not exist, make sure you run through the Step 8 in the doc you referred to.

avatar
Super Collaborator

Thanks @Robert Levas for your input I can confirm I had run through step 8 and the keytab exists:

[root@master bin]$ ls -al /etc/security/keytabs/livy.headless.keytab
-rw------- 1 livy hadoop 226 May 19 09:41 /etc/security/keytabs/livy.headless.keytab

avatar

If you klist that keytab file, does it show entries for livy/<current hostname>@LBG.COM?

klist -kte /etc/security/keytabs/livy.headless.keytab

Also, if you look at the running Livy server process, does the relevant user or group have read access to /etc/security/keytabs/livy.headless.keytab?

avatar

@David Tam

I think I found the issue after walking through the steps in the doc you provided - https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_command-line-installation/content/config....

The documentation appears to be incorrect and there is an inconsistency with the Livy Kerberos identity.

Step 8a contains

kadmin.local -q "addprinc -randkey livy@EXAMPLE.COM"
kadmin.local -q "xst -k /etc/security/keytabs/livy.headless.keytab livy@EXAMPLE.COM"

Then step 8c contains

livy.server.launch.kerberos.keytab /etc/security/keytabs/livy.headless.keytab
livy.server.launch.kerberos.principal livy/_HOST@EXAMPLE.COM

The disconnect is the principal name: livy@EXAMPLE.COM vs livy/_HOST@EXAMPLE.COM. One or the other needs to be used.

Technically, Livy Server is not an interactive user, so a service principal should be created for it. So I would choose to use the livy/_HOST@EXAMPLE.COM representation of the Kerberos identity. However it might be easier to choose the user identity and go with livy@EXAMPLE.COM. In any case, the data in the configuration needs to be consistent.

If you chose to go with the user Kerberos identity, then the following values need to be set in the livy.conf file:

livy.server.launch.kerberos.keytab /etc/security/keytabs/livy.headless.keytab
livy.server.launch.kerberos.principal livy@EXAMPLE.COM

I assume that the user principal has already been created in the KDC and the keytab file was created using it. So after restarting the Livy Server, all should work.

If you choose to go the service principal route, then the following values need to be set in the livy.conf file:

livy.server.launch.kerberos.keytab /etc/security/keytabs/livy.service.keytab
livy.server.launch.kerberos.principal livy/_HOST@EXAMPLE.COM

However, I am not sure if Livy automatically translates _HOST to the relevant host's name. So it might be safer (but less portable) to explicitly put the host name there. You can get the correct hostname by issuing a the following command

hostname -f

For example:

[root@c6403 ~]# hostname -f
c6403.ambari.apache.org

Using this value, manually replace _HOST:

livy.server.launch.kerberos.keytab /etc/security/keytabs/livy.service.keytab
livy.server.launch.kerberos.principal livy/c6403.ambari.apache.org@EXAMPLE.COM

Then you need to create the relevant Kerberos principal and keytab file,. If possible, you can do this from the Livy server host, using kadmin, rather than kadmin.local:

kadmin -p <kdc admin principal> -q "addprinc -randkey livy/`hostname -f`@EXAMPLE.COM"
kadmin -p <kdc admin principal> -q "xst -k /etc/security/keytabs/livy.service.keytab livy/`hostname -f`@EXAMPLE.COM"

For example:

[root@c6403 ~]# kadmin -p admin/admin -q "addprinc -randkey livy/`hostname -f`@EXAMPLE.COM"
Authenticating as principal admin/admin with password.
Password for admin/admin@EXAMPLE.COM:
WARNING: no policy specified for livy/c6403.ambari.apache.org@EXAMPLE.COM; defaulting to no policy
Principal "livy/c6403.ambari.apache.org@EXAMPLE.COM" created.

[root@c6403 ~]# kadmin -p admin/admin -q "xst -k /etc/security/keytabs/livy.headless.keytab livy/`hostname -f`@EXAMPLE.COM"
Authenticating as principal admin/admin with password.
Password for admin/admin@EXAMPLE.COM:
Entry for principal livy/c6403.ambari.apache.org@EXAMPLE.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/security/keytabs/livy.headless.keytab.
Entry for principal livy/c6403.ambari.apache.org@EXAMPLE.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/security/keytabs/livy.headless.keytab.
Entry for principal livy/c6403.ambari.apache.org@EXAMPLE.COM with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/etc/security/keytabs/livy.headless.keytab.
Entry for principal livy/c6403.ambari.apache.org@EXAMPLE.COM with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/etc/security/keytabs/livy.headless.keytab.
Entry for principal livy/c6403.ambari.apache.org@EXAMPLE.COM with kvno 2, encryption type des-hmac-sha1 added to keytab WRFILE:/etc/security/keytabs/livy.headless.keytab.
Entry for principal livy/c6403.ambari.apache.org@EXAMPLE.COM with kvno 2, encryption type des-cbc-md5 added to keytab WRFILE:/etc/security/keytabs/livy.headless.keytab.

If you need to use kadmin.local on the KDC server host, you can do the following the manually copy the resulting keytab file into place on the Livy server host:

kadmin.local -q "addprinc -randkey livy/`hostname -f`@EXAMPLE.COM"
kadmin.local -q "xst -k /etc/security/keytabs/livy.service.keytab livy/`hostname -f`@EXAMPLE.COM"

Meanwhile I will see if the documentation can be fixed.

avatar
Super Collaborator

hello @Robert Levas thank you for the detailed explanation.

I tried what you said, but still getting the same error:

[root@master livy]# whoami
root
[root@master livy]# hostname -f
master.sandbox.lbg.com
[root@master livy]# kadmin.local -q "addprinc -randkey livy/master.sandbox.lbg.com@LBG.COM"
Authenticating as principal root/admin@LBG.COM with password.
WARNING: no policy specified for livy/master.sandbox.lbg.com@LBG.COM; defaulting to no policy
Principal "livy/master.sandbox.lbg.com@LBG.COM" created.
[root@master livy]# kadmin.local -q "xst -k /etc/security/keytabs/livy.headless.keytab livy/master.sandbox.lbg.com@LBG.COM"
Authenticating as principal root/admin@LBG.COM with password.
Entry for principal livy/master.sandbox.lbg.com@LBG.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/etc/security/keytabs/livy.headless.keytab.
Entry for principal livy/master.sandbox.lbg.com@LBG.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/etc/security/keytabs/livy.headless.keytab.
Entry for principal livy/master.sandbox.lbg.com@LBG.COM with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/etc/security/keytabs/livy.headless.keytab.
Entry for principal livy/master.sandbox.lbg.com@LBG.COM with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/etc/security/keytabs/livy.headless.keytab.
[root@master livy]#
[root@master livy]# cat /etc/livy/conf/livy.conf
livy.spark.master = yarn
livy.spark.deployMode = cluster
livy.environment production
livy.impersonation.enabled true
livy.server.csrf_protection.enabled true
livy.server.port 8998
livy.server.session.timeout 3600000
livy.server.recovery.mode off

livy.server.auth.type = kerberos
livy.server.launch.kerberos.keytab = /etc/security/keytabs/livy.headless.keytab
livy.server.launch.kerberos.principal = livy/master.sandbox.lbg.com@LBG.COM

livy.server.kerberos.keytab = /etc/security/keytabs/livy.headless.keytab
livy.server.auth.kerberos.keytab /etc/security/keytabs/spnego.service.keytab
livy.server.auth.kerberos.principal HTTP/_HOST@LBG.COM

livy.superusers=livy

Then:

[livy@master bin]$ whoami
livy
[livy@master bin]$ hostname -f
master.sandbox.lbg.com
[livy@master bin]$ ls -al /etc/security/keytabs/livy.headless.keytab
-rw------- 1 livy hadoop 546 May 22 08:59 /etc/security/keytabs/livy.headless.keytab
[livy@master bin]$ klist -kte /etc/security/keytabs/livy.headless.keytab
Keytab name: FILE:/etc/security/keytabs/livy.headless.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   2 05/19/17 09:41:41 livy@LBG.COM (aes256-cts-hmac-sha1-96)
   2 05/19/17 09:41:41 livy@LBG.COM (aes128-cts-hmac-sha1-96)
   2 05/19/17 09:41:41 livy@LBG.COM (des3-cbc-sha1)
   2 05/19/17 09:41:41 livy@LBG.COM (arcfour-hmac)
   2 05/22/17 08:59:56 livy/master.sandbox.lbg.com@LBG.COM (aes256-cts-hmac-sha1-96)
   2 05/22/17 08:59:56 livy/master.sandbox.lbg.com@LBG.COM (aes128-cts-hmac-sha1-96)
   2 05/22/17 08:59:56 livy/master.sandbox.lbg.com@LBG.COM (des3-cbc-sha1)
   2 05/22/17 08:59:56 livy/master.sandbox.lbg.com@LBG.COM (arcfour-hmac)
[livy@master bin]$ /usr/hdp/current/livy-server/bin/livy-server start
starting /usr/java/default/bin/java -Xmx2g -cp /usr/hdp/current/livy-server/jars/*:/usr/hdp/current/livy-server/conf: com.cloudera.livy.server.LivyServer, logging to /var/log/livy/livy-livy-server.out
[livy@master bin]$ cat /var/log/livy/livy-livy-server.out
log4j:WARN No appenders could be found for logger (com.cloudera.livy.server.LivyServer).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See <a href="http://logging.apache.org/log4j/1.2/faq.html#noconfig">http://logging.apache.org/log4j/1.2/faq.html#noconfig</a> for more info.
Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: Kerberos requires livy.server.kerberos.keytab to be provided.
        at scala.Predef$.require(Predef.scala:233)
        at com.cloudera.livy.server.LivyServer.runKinit(LivyServer.scala:173)
        at com.cloudera.livy.server.LivyServer.start(LivyServer.scala:134)
        at com.cloudera.livy.server.LivyServer$.main(LivyServer.scala:277)
        at com.cloudera.livy.server.LivyServer.main(LivyServer.scala)
[livy@master root]$ /usr/hdp/current/livy-server/bin/livy-server stop
no livy_server to stop

So if you did all you said on your box you were able to start up livy without this issue?

avatar

I think that I was able to get it running. I started the server and did not see the error message you are getting.

One thing that I see is that the server may be looking for a file named "livy.server.kerberos.keytab" or maybe it cannot find that property. Is it possible a different configuration file is being picked up?

avatar
Super Collaborator

Hello @Robert Levas thank you for your reply.

I am pretty sure the conf file is being picked up. If I comment out this line:

livy.server.auth.type = kerberos

Then the server can start up fine, and requests are served fine, just that without authentication.

Adding or removing the following has no effect:

livy.server.kerberos.keytab = /etc/security/keytabs/livy.headless.keytab

even though the log kind of suggest it is looking for it, if kerberos is switched on.