Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Who agreed with this solution

Re: Zeppelin, Livy, Hive, Kerberos & Spark 1.6

New Contributor

Solved this.


Hive from Zeppelin w. Livy & Spark2


Add spark2 to the kerberized cluster


Adapt livy to recognize spark2 if required


#build livy for spark2
mvn clean package -DskipTests -Dspark-2.0 -Dscala-2.11

#make livy hive aware
sudo ln -s /etc/hive/conf/hive-site.xml /etc/spark/conf/hive-site.xml

#make a livy user
sudo useradd livy

#create a logs dir
mkdir logs

#change the ownership of everything
sudo chown -R livy:livy ./*

#start livy with the right config (see below)
sudo -u livy /opt/livy/bin/livy-server
export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2
export HADOOP_HOME=/opt/cloudera/parcels/CDH
export SPARK_CONF_DIR=/etc/spark/conf
export HADOOP_CONF_DIR=/etc/hadoop/conf

# What spark master Livy sessions should use.
livy.spark.master = yarn

# What spark deploy mode Livy sessions should use.
livy.spark.deployMode = cluster

# If livy should impersonate the requesting users when creating a new session.
livy.impersonation.enabled = true

# Whether to enable HiveContext in livy interpreter, if it is true hive-site.xml will be detected
# on user request and then livy server classpath automatically.
livy.repl.enableHiveContext = true

livy.server.launch.kerberos.keytab = /opt/livy/livy.keytab

livy.impersonation.enabled = true
livy.server.auth.type = kerberos





You must also configure Zeppelin for kerberos auth:

zeppelin.livy.keytab	/opt/zeppelin/zeppelin.keytab
zeppelin.livy.principal	zeppelin@XXX
zeppelin.livy.url	http://host.fqdn:8998

Finally you must configure shiro, I used an LDAP backend with org.apache.zeppelin.realm.LdapGroupRealm plugin to enable LDAP user group awareness (take care to set up groupOfNames not posixGroup...).





Who agreed with this solution