Problem with Zeppelin, Spark+Livy and LLAP in Kerberos HDP

[#1]

Hi all

So, I have a problem getting Spark to work with LLAP in Zeppelin with the help of Livy. The version I’m using is HDP 2.6.1, but I had the same problem with 2.6.0. I’ve looked and followed the two guides in https://community.hortonworks.com/articles/110093/using-rowcolumn-level-security-of-spark-with-zeppe... and https://community.hortonworks.com/content/kbentry/101181/rowcolumn-level-security-in-sql-for-apache-..., but still no luck.

Spark (no SQL/LLAP) with Livy works fine in Zeppelin together with Kerberos. I can submit jobs in Zeppelin, and they gets executed in Yarn correctly, with the right user and everything. So I know that the Kerberos configuration is correct and working. But as soon as I try to run a Spark SQL code, I get a Kerberos error saying that I don’t have a valid Kerberos ticket. (Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt))

Normal Spark without Livy works fine with LLAP. No problem there at all.

Configuration that I’m running with right now.

Spark is configured with the following settings

spark.hadoop.hive.llap.daemon.service.hosts @llap0
spark.hadoop.hive.zookeeper.quorum <3xZK servers + ports>
spark.sql.hive.hiveserver2.jdbc.url jdbc:hive2://<server>:10500/
spark.sql.hive.hiveserver2.jdbc.url.principal <HIVE principle>
spark.sql.hive.llap true

Livy server is configured with the following settings

livy.impersonation.enabled true
livy.repl.enableHiveContext true
livy.server.access_control.enabled true
livy.server.access_control.users livy,zeppelin
livy.server.auth.kerberos.keytab <SPNEGO keytab>
livy.server.auth.kerberos.principal <SPNEGO principle>
livy.server.auth.type Kerberos
livy.server.launch.kerberos.keytab <LIVY keytab>
livy.server.launch.kerberos.principal <LIVY principle>
livy.superusers livy,zeppelin

Zeppelin interpreter is configured with the following settings

livy.spark.hadoop.hive.llap.daemon.service.hosts @llap0
livy.spark.jars /lib/spark-llap_2.11-1.1.2-2.1.jar (HDFS file)
livy.spark.sql.hive.hiveserver2.jdbc.url jdbc:hive2://<server>:10500/
livy.spark.sql.hive.hiveserver2.jdbc.url.principal <HIVE principle>
livy.spark.sql.hive.llap true
livy.superusers livy,zeppelin
zeppelin.livy.keytab <LIVY keytab>
zeppelin.livy.principal <LIVY principle>
zeppelin.livy.url http://<server>:8999

Is there any other configuration, except the once that are already described in the articles above, that I need to make to be able to get a valid Kerberos ticket from the Spark session that Livy creates for me? Or do you have any more information that can help me to get this functionality to work?

Best Regards
Berry Österlund

[#2]

I have found the solution to my problem. Add the following to the Livy interpreter in Zeppelin. Once that is done, I can use Spark SQL with LLAP and Livy in Zeppelin. The jobs are executed as the logged in Zeppelin users (Shiro authentication to Microsoft AD) and that is verified in Hive2's history log.

livy.spark.yarn.security.credentials.hiveserver2.enabled true