How to configure zeppelin livy interpreter for secure HDP cluster

[#1]

Livy: Livy is an open source REST interface for interacting with Spark. Authorized users can launch a Spark session and submit code. Two different users can access their own private data and session, and they can collaborate on a notebook. Only the Livy server can submit a job securely to a Spark session.

Steps to follow to configure livy interpreter to work with secure HDP cluster:

1. Setup proxy for livy interpreter in core-site.xml

Go to Ambari->HDFS->config->customer-core-site and add below properties: 

hadoop.proxyuser.livy.groups=* 

hadoop.proxyuser.livy.hosts=*

2. Configure livy interpreter in Zeppelin and add below configurations:

livy.superusers=zeppelin-spark

Note - The value for livy.superusers should be your zeppelin principal. That would be zeppelin-{$Cluster_name} For example, in this case you can find it by running below command:

klist -kt /etc/security/keytabs/zeppelin.server.kerberos.keytab

Keytab name: FILE:/etc/security/keytabs/zeppelin.server.kerberos.keytab
KVNO Timestamp         Principal
---- ----------------- --------------------------------------------------------
   1 11/15/16 17:33:16 zeppelin-spark@HWX.COM
   1 11/15/16 17:33:16 zeppelin-spark@HWX.COM
   1 11/15/16 17:33:16 zeppelin-spark@HWX.COM
   1 11/15/16 17:33:16 zeppelin-spark@HWX.COM
   1 11/15/16 17:33:16 zeppelin-spark@HWX.COM

zeppelin-spark will be your superuser for livy interpreter.

*Make sure this will match with livy.superusers in livy-conf file.

livy.impersonation.enabled=true  //this configuration should also be present in livy-conf. 
livy.server.access_control.enabled=true 
livy.server.access_control.users=livy,zeppelin 
livy.server.auth.type=kerberos 
livy.server.auth.kerberos.keytab=/etc/security/keytabs/spnego.service.keytab 
livy.server.auth.kerberos.principal=HTTP/spark-1.hwx.com@HWX.COM 
livy.server.launch.kerberos.keytab=/etc/security/keytabs/livy.service.keytab 
livy.server.launch.kerberos.principal=livy/spark-1.hwx.com@HWX.COM

Note - To configure Zeppelin with authentication for Livy you need to set the following in the interpreter settings:

zeppelin.livy.principal=zeppelin-spark@HWX.COM 
zeppelin.livy.keytab=/etc/security/keytabs/zeppelin.service.keytab

3. Make sure zeppelin.livy.url is pointing to hostname not IP address :

zeppelin.livy.url=http://spark-3.hwx.com:8998

4. After saving configuration changes in livy interpreter, Please restart interpreter to see the affect.