Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Zeppelin livy interpreter not working with Kerberos cluster on HDP 2.5.3

Solved Go to solution
Highlighted

Re: Zeppelin livy interpreter not working with Kerberos cluster on HDP 2.5.3

Cloudera Employee

I too had run into several of the issues above but with 2.5.0. The 2 major things that I change on top of what is listed above is:

Add these items in Ambari:

livy.conf

livy.server.launch.kerberos.keytab /etc/security/keytabs/livy.service.keytab

livy.server.launch.kerberos.principal livy/_HOST@REALM

livy-env.sh

export HADOOP_HOME=/usr/hdp/current/hadoop-client

I was getting 401 errors from Zeppelin to Livy:

WARN [2017-01-06 11:07:29,393] ({pool-2-thread-4} HttpAuthenticator.java[generateAuthResponse]:207) - NEGOTIATE authentication error: No valid credentials provided (Mechani sm level: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER))

ERROR [2017-01-06 11:07:29,407] ({pool-2-thread-4} LivyHelper.java[createSession]:128) - Error getting session for user org.springframework.web.client.RestClientException: Error running rest call; nested exception is org.springframework.web.client.HttpClientErrorException: 401 Authentication required

Highlighted

Re: Zeppelin livy interpreter not working with Kerberos cluster on HDP 2.5.3

I don't really have an answer but I do have some more information.

I see that zeppelin contacts livy, authenticates successfully and that livy replies with a:

Set-Cookie: hadoop.auth="u=zeppelin..."; HttpOnly

However I don't ever see that cookie returned.

As far as I can see the livy interpreter should send the cookie on every call to livy. It uses org.springframework.web.client.RestTemplate to handle communication between itself and the livy server and I can see that that framework can handle cookies but I also see that the cookie is missing.

Highlighted

Re: Zeppelin livy interpreter not working with Kerberos cluster on HDP 2.5.3

New Contributor

Hello,

Did you figured out the solution?

I'm working on similar configuration since couple weeks and I'm still getting error:

%livy.spark
sc.version
Cannot start  spark. 

I have debug enabled on livy but don't see any specific error.

In zeppelin-interpreter-livy log I see:

ERROR [2017-02-03 15:07:47,491] ({pool-2-thread-7} LivyHelper.java[createSession]:128) - Error getting session for user
java.lang.Exception: Cannot start  spark. 

What's interesting, I see spark job created in yarn history with my user over livy-session with the status finished/suceeded most of the time, so from that side it looks correct. Unfortunatelly, still no result in zeppelin notebook.

I will really appreciate any suggestions from those who succeeded:)

Kind regards, Michał

Re: Zeppelin livy interpreter not working with Kerberos cluster on HDP 2.5.3

Expert Contributor

Michal, my problem was related to KMS proxy user and addressed by adding livy as proxy user. My cluster was secured with Kerberos and with Ranger and KMS. I am not sure if your is similar. You may want to start a new thread with more details.

Highlighted

Re: Zeppelin livy interpreter not working with Kerberos cluster on HDP 2.5.3

The problem isn't in zeppelin, it is in livy. Check livy.out you may see a timeout connecting to hive.

Highlighted

Re: Zeppelin livy interpreter not working with Kerberos cluster on HDP 2.5.3

I now have a working livy running, at least sc.version works

After trying everything I could find with livy 0.2.0 (the version in 2.5.0) I decided to give 0.3.0 a try. I believe that the problem is caused by a bug in spark 1.6.2 when connecting to the metadata store.

After compiling livy 0.3.0 with hadoop 2.7.3 and spark 2.0.0, and installing it beside 0.2.0 I had problems creating credentials for the HTTP principal. I solved that by using the hadoop jars from livy 0.2.0 instead of those from the build.

Don't have an account?
Coming from Hortonworks? Activate your account here