Reply
Highlighted
New Contributor
Posts: 2
Registered: ‎01-12-2017

How to configure Livy impersonation for Spark / Zeppelin

My end goal is to be able to run Zeppelin/Livy/Spark with impersonation. So far, I have successfully configured Zeppelin with Spark. However, I want to use multi-tenancy, and for that I wanted to configure Zeppelin with Livy and Spark.

 

For Livy, I provided the following two paths

export SPARK_HOME=/opt/cloudera/parcels/CDH/lib/spark
export HADOOP_CONF_DIR=/etc/hadoop/conf

With the above settings, I can run the following command successfully in Zeppelin:

%livy.spark
sc.version

However, the following command fails:

%livy.sql
select * from myDB.table1

I see the following error:

<console>:14: error: not found: value sqlContext
              sqlContext.sql("select * from datalake.combination2").show(1000)

I have not enabled Shiro authentication for Zeppelin yet. My assumption was that Livy would log into Spark using the default user as I provide the Spark home directory. Could anyone point out how can I fix the above issue? And also, what Livy/Spark configurations would I need to do to enable impersonation once I enable the Shiro authentication for Zeppelin?

Cloudera Employee
Posts: 20
Registered: ‎01-17-2017

Re: How to configure Livy impersonation for Spark / Zeppelin

-# Find the java binary
-if [ -n "${JAVA_HOME}" ]; then
- RUNNER="${JAVA_HOME}/bin/java"
-elif [ `command -v java` ]; then
- RUNNER="java"
-else
- echo "JAVA_HOME is not set" >&2
- exit 1
-fi

New Contributor
Posts: 2
Registered: ‎01-12-2017

Re: How to configure Livy impersonation for Spark / Zeppelin

@ZachRoes Thanks for your reply. I located java installation, and ran

 

export RUNNER=/usr/bin/java

 

I then started Livy, and restarted livy interpreter in Zeppelin, but I still see error in Zeppelin. Is there anything else that I might be missing? As a first step, I just want to run livy.sql in Zeppelin without impersonation.