So, I have a problem getting Spark to work with LLAP in Zeppelin with the help of Livy. The version I’m using is HDP 2.6.1, but I had the same problem with 2.6.0. I’ve looked and followed the two guides in https://community.hortonworks.com/articles/110093/using-rowcolumn-level-security-of-spark-with-zeppe... and https://community.hortonworks.com/content/kbentry/101181/rowcolumn-level-security-in-sql-for-apache-..., but still no luck.
Spark (no SQL/LLAP) with Livy works fine in Zeppelin together with Kerberos. I can submit jobs in Zeppelin, and they gets executed in Yarn correctly, with the right user and everything. So I know that the Kerberos configuration is correct and working. But as soon as I try to run a Spark SQL code, I get a Kerberos error saying that I don’t have a valid Kerberos ticket. (Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt))
Normal Spark without Livy works fine with LLAP. No problem there at all.
Configuration that I’m running with right now.
Spark is configured with the following settings
spark.hadoop.hive.zookeeper.quorum <3xZK servers + ports>
spark.sql.hive.hiveserver2.jdbc.url.principal <HIVE principle>
Livy server is configured with the following settings
livy.server.auth.kerberos.keytab <SPNEGO keytab>
livy.server.auth.kerberos.principal <SPNEGO principle>
livy.server.launch.kerberos.keytab <LIVY keytab>
livy.server.launch.kerberos.principal <LIVY principle>
Zeppelin interpreter is configured with the following settings
livy.spark.jars /lib/spark-llap_2.11-1.1.2-2.1.jar (HDFS file)
livy.spark.sql.hive.hiveserver2.jdbc.url.principal <HIVE principle>
zeppelin.livy.keytab <LIVY keytab>
zeppelin.livy.principal <LIVY principle>
Is there any other configuration, except the once that are already described in the articles above, that I need to make to be able to get a valid Kerberos ticket from the Spark session that Livy creates for me? Or do you have any more information that can help me to get this functionality to work?
I have found the solution to my problem. Add the following to the Livy interpreter in Zeppelin. Once that is done, I can use Spark SQL with LLAP and Livy in Zeppelin. The jobs are executed as the logged in Zeppelin users (Shiro authentication to Microsoft AD) and that is verified in Hive2's history log.