Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Who agreed with this solution

Re: Zeppelin, Livy, Hive, Kerberos & Spark 1.6

New Contributor

Solved this.

 

Hive from Zeppelin w. Livy & Spark2

 

Add spark2 to the kerberized cluster

https://www.cloudera.com/downloads/spark2/2-0.html

 

https://www.cloudera.com/documentation/enterprise/latest/topics/cm_mc_addon_services.html#concept_qb...

 

Adapt livy to recognize spark2 if required

https://community.cloudera.com/t5/Web-UI-Hue-Beeswax/Spark-2-0-livy-server-3/td-p/48562

 

#build livy for spark2
mvn clean package -DskipTests -Dspark-2.0 -Dscala-2.11

#make livy hive aware
sudo ln -s /etc/hive/conf/hive-site.xml /etc/spark/conf/hive-site.xml

#make a livy user
sudo useradd livy

#create a logs dir
mkdir logs

#change the ownership of everything
sudo chown -R livy:livy ./*

#start livy with the right config (see below)
sudo -u livy /opt/livy/bin/livy-server

livy-env.sh
===============================================
export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2
export HADOOP_HOME=/opt/cloudera/parcels/CDH
export SPARK_CONF_DIR=/etc/spark/conf
export HADOOP_CONF_DIR=/etc/hadoop/conf

livy.conf
===============================================
# What spark master Livy sessions should use.
livy.spark.master = yarn

# What spark deploy mode Livy sessions should use.
livy.spark.deployMode = cluster

# If livy should impersonate the requesting users when creating a new session.
livy.impersonation.enabled = true

# Whether to enable HiveContext in livy interpreter, if it is true hive-site.xml will be detected
# on user request and then livy server classpath automatically.
livy.repl.enableHiveContext = true

livy.server.launch.kerberos.keytab = /opt/livy/livy.keytab
livy.server.launch.kerberos.principal=livy/server.fqdn@XXX

livy.impersonation.enabled = true
livy.server.auth.type = kerberos

livy.server.auth.kerberos.keytab=/opt/livy/spnego.keytab
livy.server.auth.kerberos.principal=HTTP/server.fqdn@XXX

livy.server.access_control.enabled=true
livy.server.access_control.users=zeppelin,livy

livy.superusers=zeppelin,livy

 

You must also configure Zeppelin for kerberos auth:

zeppelin.livy.keytab	/opt/zeppelin/zeppelin.keytab
zeppelin.livy.principal	zeppelin@XXX
zeppelin.livy.url	http://host.fqdn:8998

Finally you must configure shiro, I used an LDAP backend with org.apache.zeppelin.realm.LdapGroupRealm plugin to enable LDAP user group awareness (take care to set up groupOfNames not posixGroup...).

 

 

 

 

Who agreed with this solution