Livy Interpreter: Cannot start spark

[#1]

This is applicable in a Kerberos enabled HDP 2.5.x cluster with Zeppelin, Livy & Spark.

Post successful Kerberos setup, log in to Zeppelin and run Spark note, the note runs file.

But running simple sc.version from livy interpreter gives "Cannot start spark" in the Zeppelin UI.

In the Livy log at /var/log/livy/livy-livy-server.out you may find a message similar to the following.

INFO: 17/05/25 21:24:12 INFO metastore: Trying to connect to metastore with URI thrift://vinay-hdp25-2.field.hortonworks.com:9083

May 25, 2017 9:24:12 PM org.apache.spark.launcher.OutputRedirector redirect

INFO: 17/05/25 21:24:12 ERROR TSaslTransport: SASL negotiation failure

May 25, 2017 9:24:12 PM org.apache.spark.launcher.OutputRedirector redirect

INFO: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]

This happens when Livy tried to connect to Hive Metastore and fails with above message.

The fix is to configure Zeppelin's Livy interpreter to run in yarn-cluster mode, instead of the default yarn-client mode. After you change any interpreter configuration, you will need to restart the interpreter.

Below works.

livy.spark.master

yarn-cluster

Starting HDP 2.6.x this configuration is changed OOB to yarn-cluster.