Hi all
So, I have a problem getting Spark to work with LLAP in
Zeppelin with the help of Livy. The version I’m using is HDP 2.6.1, but I had
the same problem with 2.6.0. I’ve looked and followed the two guides in https://community.hortonworks.com/articles/110093/using-rowcolumn-level-security-of-spark-with-zeppe...
and https://community.hortonworks.com/content/kbentry/101181/rowcolumn-level-security-in-sql-for-apache-...,
but still no luck.
Spark (no SQL/LLAP) with Livy works fine in Zeppelin
together with Kerberos. I can submit jobs in Zeppelin, and they gets executed
in Yarn correctly, with the right user and everything. So I know that the Kerberos
configuration is correct and working. But as soon as I try to run a Spark SQL
code, I get a Kerberos error saying that I don’t have a valid Kerberos ticket. (Caused by: GSSException: No valid credentials
provided (Mechanism level: Failed to find any Kerberos tgt))
Normal Spark without Livy works fine with LLAP. No problem
there at all.
Configuration that I’m running with right now.
Spark is configured with the following settings
spark.hadoop.hive.llap.daemon.service.hosts @llap0
spark.hadoop.hive.zookeeper.quorum <3xZK servers
+ ports>
spark.sql.hive.hiveserver2.jdbc.url jdbc:hive2://<server>:10500/
spark.sql.hive.hiveserver2.jdbc.url.principal <HIVE principle>
spark.sql.hive.llap true
Livy server is configured with the following settings
livy.impersonation.enabled true
livy.repl.enableHiveContext true
livy.server.access_control.enabled true
livy.server.access_control.users livy,zeppelin
livy.server.auth.kerberos.keytab <SPNEGO
keytab>
livy.server.auth.kerberos.principal <SPNEGO
principle>
livy.server.auth.type Kerberos
livy.server.launch.kerberos.keytab <LIVY
keytab>
livy.server.launch.kerberos.principal <LIVY
principle>
livy.superusers livy,zeppelin
Zeppelin interpreter is
configured with the following settings
livy.spark.hadoop.hive.llap.daemon.service.hosts @llap0
livy.spark.jars /lib/spark-llap_2.11-1.1.2-2.1.jar
(HDFS file)
livy.spark.sql.hive.hiveserver2.jdbc.url jdbc:hive2://<server>:10500/
livy.spark.sql.hive.hiveserver2.jdbc.url.principal <HIVE principle>
livy.spark.sql.hive.llap true
livy.superusers livy,zeppelin
zeppelin.livy.keytab <LIVY
keytab>
zeppelin.livy.principal <LIVY principle>
zeppelin.livy.url http://<server>:8999
Is there any other configuration, except the once that are
already described in the articles above, that I need to make to be able to get
a valid Kerberos ticket from the Spark session that Livy creates for me? Or do
you have any more information that can help me to get this functionality to
work?
Best Regards
Berry Österlund
