Livy: Livy is an open source REST interface for interacting with Spark. Authorized users can launch a Spark session and submit code. Two different users can access their own private data and session, and they can collaborate on a notebook. Only the Livy server can submit a job securely to a Spark session. Steps to follow to configure livy interpreter to work with secure HDP cluster: Setup proxy for livy interpreter in core-site.xml Go to Ambari->HDFS->config->customer-core-site and add below properties:
hadoop.proxyuser.livy.groups=*
hadoop.proxyuser.livy.hosts=*
2. Configure livy interpreter in Zeppelin and add below configurations: livy.superusers=zeppelin-spark
Note - The value for livy.superusers should be your zeppelin principal. That would be zeppelin-{$Cluster_name} For example, in this case you can find it by running below command:
klist -kt /etc/security/keytabs/zeppelin.server.kerberos.keytab
Keytab name: FILE:/etc/security/keytabs/zeppelin.server.kerberos.keytab
KVNO Timestamp Principal
---- ----------------- --------------------------------------------------------
1 11/15/16 17:33:16 zeppelin-spark@HWX.COM
1 11/15/16 17:33:16 zeppelin-spark@HWX.COM
1 11/15/16 17:33:16 zeppelin-spark@HWX.COM
1 11/15/16 17:33:16 zeppelin-spark@HWX.COM
1 11/15/16 17:33:16 zeppelin-spark@HWX.COM
zeppelin-spark will be your superuser for livy interpreter. *Make sure this will match with livy.superusers in livy-conf file. livy.impersonation.enabled=true //this configuration should also be present in livy-conf.
livy.server.access_control.enabled=true
livy.server.access_control.users=livy,zeppelin
livy.server.auth.type=kerberos
livy.server.auth.kerberos.keytab=/etc/security/keytabs/spnego.service.keytab
livy.server.auth.kerberos.principal=HTTP/spark-1.hwx.com@HWX.COM
livy.server.launch.kerberos.keytab=/etc/security/keytabs/livy.service.keytab
livy.server.launch.kerberos.principal=livy/spark-1.hwx.com@HWX.COM
Note - To configure Zeppelin with authentication for Livy you need to set the following in the interpreter settings: zeppelin.livy.principal=zeppelin-spark@HWX.COM
zeppelin.livy.keytab=/etc/security/keytabs/zeppelin.service.keytab
3. Make sure zeppelin.livy.url is pointing to hostname not IP address : zeppelin.livy.url=http://spark-3.hwx.com:8998 4. After saving configuration changes in livy interpreter, Please restart interpreter to see the affect.
... View more
