Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Zeppelin with kerberized Livy not working

avatar
New Contributor

I have Zeppelin 0.6.2 pre-built binary with Kerberised Livy 0.2 on a Kerberized Hadoop/Spark cluster. When I use the Livy interpreter with Spark, I hit the below error in the zeppelin logs (the livy logs do not show anything as the connection is not hitting Livy due to a failure in zeppelin)

INFO [2017-01-06 12:53:05,121] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:131) - Job remoteInterpretJob_1483707185119 started by scheduler org.apache.zeppelin.livy.LivySparkInterpreter1764226124 WARN [2017-01-06 12:53:05,891] ({pool-2-thread-2} HttpAuthenticator.java[generateAuthResponse]:207) - NEGOTIATE authentication error: No valid credentials provided (Mechanism level: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7))) ERROR [2017-01-06 12:53:05,917] ({pool-2-thread-2} LivyHelper.java[createSession]:121) - Error getting session for user org.springframework.web.client.RestClientException: Error running rest call; nested exception is org.springframework.web.client.HttpClientErrorException: 401 Authentication required at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:196) at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:580) at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:498) at org.apache.zeppelin.livy.LivyHelper.executeHTTP(LivyHelper.java:344) at org.apache.zeppelin.livy.LivyHelper.createSession(LivyHelper.java:73) at org.apache.zeppelin.livy.LivySparkInterpreter.interpret(LivySparkInterpreter.java:101) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:341) at org.apache.zeppelin.scheduler.Job.run(Job.java:176) at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.springframework.web.client.HttpClientErrorException: 401 Authentication required at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91) at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:667) at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:620) at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecuteSubject(KerberosRestTemplate.java:202) at org.springframework.security.kerberos.client.KerberosRestTemplate.access$100(KerberosRestTemplate.java:67) at org.springframework.security.kerberos.client.KerberosRestTemplate$1.run(KerberosRestTemplate.java:191) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:187)

I have set the below in the Livy interpreter:

zeppelin.livy.keytab = /home/zeppelin/zeppelin.keytab

zeppelin.livy.principal = zeppelin/host.com@DOMAIN.COM

livy.spark.master = yarn-cluster (tried with blank as well)

Zeppelin itself has authentication enabled (Shiro with Active Directory) which is working fine and lets me login using my Active Directory credentials. Kerberized Livy also works fine with CURL when using 'negotiate' after kinit. When I kinit using the zeppelin keytab (zeppelin/host.com@DOMAIN.COM), it works fine as well (rules out keytab issues) and I am able to succesfully submit CURL negotiate GET & PUT requsts to Livy. Within Livy, zeppelin is set as a super user. Any pointers would be much appreciated.

1 ACCEPTED SOLUTION

avatar
New Contributor

Fixed this by leaving livy.spark.master blank (instead of yarn-cluster) as this value is black-listed in Livy.

View solution in original post

6 REPLIES 6

avatar
Super Collaborator
@Prajwal Kumar

Can you please verify if you have hadoop proxyuser set for livy in core-site.xml.

And also the error show "Server not found in Kerberos database" and the principal used is "zeppelin/host.com@DOMAIN.COM" which doesnt seems to be correct. /home/zeppelin/zeppelin.keytab keytab mentioned in interpreter must me a user principal. Something like "zeppelin-hdp25@RAGHAV.COM".

Create keytab with a user principal and use that in your interpreter. And also make sure below properties are set

<property>
  <name>hadoop.proxyuser.livy.groups</name>
  <value>*</value>
</property>
<property>
  <name>hadoop.proxyuser.livy.hosts</name>
  <value>*</value>
</property>

avatar
New Contributor

@rguruvannagari

Thanks for the hint. Livy proxy user was already set. I have updated my Livy principal to xyzuser@DOMAIN.COM with the xyzuser being set as a super user in Livy. I have also successfully tested submission of jobs to Livy with cURL. But I still get an error in Zeppelin (this time error 400). Any pointers?

INFO [2017-01-09 01:08:29,805] ({pool-2-thread-4} SchedulerFactory.java[jobStarted]:131) - Job remoteInterpretJob_1483920509805 started by scheduler org.apache.zeppelin.livy.LivySparkInterpreter279557022 ERROR [2017-01-09 01:08:29,981] ({pool-2-thread-4} LivyHelper.java[createSession]:121) - Error getting session for user org.springframework.web.client.RestClientException: Error running rest call; nested exception is org.springframework.web.client.HttpClientErrorException: 400 Bad Request at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:196) at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:580) at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:498) at org.apache.zeppelin.livy.LivyHelper.executeHTTP(LivyHelper.java:344) at org.apache.zeppelin.livy.LivyHelper.createSession(LivyHelper.java:73) at org.apache.zeppelin.livy.LivySparkInterpreter.interpret(LivySparkInterpreter.java:101) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:341) at org.apache.zeppelin.scheduler.Job.run(Job.java:176) at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.springframework.web.client.HttpClientErrorException: 400 Bad Request at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91) at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:667) at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:620) at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecuteSubject(KerberosRestTemplate.java:202) at org.springframework.security.kerberos.client.KerberosRestTemplate.access$100(KerberosRestTemplate.java:67) at org.springframework.security.kerberos.client.KerberosRestTemplate$1.run(KerberosRestTemplate.java:191) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:187) ... 16 more ERROR [2017-01-09 01:08:29,983] ({pool-2-thread-4} LivySparkInterpreter.java[interpret]:108) - Exception in LivySparkInterpreter while interpret org.springframework.web.client.RestClientException: Error running rest call; nested exception is org.springframework.web.client.HttpClientErrorException: 400 Bad Request at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:196) at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:580) at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:498) at org.apache.zeppelin.livy.LivyHelper.executeHTTP(LivyHelper.java:344) at org.apache.zeppelin.livy.LivyHelper.createSession(LivyHelper.java:73) at org.apache.zeppelin.livy.LivySparkInterpreter.interpret(LivySparkInterpreter.java:101) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:341) at org.apache.zeppelin.scheduler.Job.run(Job.java:176) at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.springframework.web.client.HttpClientErrorException: 400 Bad Request at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91) at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:667) at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:620) at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecuteSubject(KerberosRestTemplate.java:202) at org.springframework.security.kerberos.client.KerberosRestTemplate.access$100(KerberosRestTemplate.java:67) at org.springframework.security.kerberos.client.KerberosRestTemplate$1.run(KerberosRestTemplate.java:191) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:187) ... 16 more

avatar
New Contributor

Fixed this by leaving livy.spark.master blank (instead of yarn-cluster) as this value is black-listed in Livy.

avatar
Expert Contributor

@Prajwal Kumar Did not work for me.

Removing it started giving new errors :

ERROR [2017-01-24 15:10:47,037] ({pool-2-thread-2} LivyHelper.java[createSession]:128) - Error getting session for user java.lang.Exception: Cannot start spark. at org.apache.zeppelin.livy.LivyHelper.createSession(LivyHelper.java:117) at org.apache.zeppelin.livy.LivySparkInterpreter.interpret(LivySparkInterpreter.java:101)

avatar
Contributor

I believe that the problem is that localhost is not a valid host for kerberos. I changed my livy interpreter settings to point out the actual host.

avatar

Can you try livy.spark.master as YARN