Zeppelin with kerberized Livy not working

[#1]

I have Zeppelin 0.6.2 pre-built binary with Kerberised Livy 0.2 on a Kerberized Hadoop/Spark cluster. When I use the Livy interpreter with Spark, I hit the below error in the zeppelin logs (the livy logs do not show anything as the connection is not hitting Livy due to a failure in zeppelin)

INFO [2017-01-06 12:53:05,121] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:131) - Job remoteInterpretJob_1483707185119 started by scheduler org.apache.zeppelin.livy.LivySparkInterpreter1764226124 WARN [2017-01-06 12:53:05,891] ({pool-2-thread-2} HttpAuthenticator.java[generateAuthResponse]:207) - NEGOTIATE authentication error: No valid credentials provided (Mechanism level: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7))) ERROR [2017-01-06 12:53:05,917] ({pool-2-thread-2} LivyHelper.java[createSession]:121) - Error getting session for user org.springframework.web.client.RestClientException: Error running rest call; nested exception is org.springframework.web.client.HttpClientErrorException: 401 Authentication required at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:196) at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:580) at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:498) at org.apache.zeppelin.livy.LivyHelper.executeHTTP(LivyHelper.java:344) at org.apache.zeppelin.livy.LivyHelper.createSession(LivyHelper.java:73) at org.apache.zeppelin.livy.LivySparkInterpreter.interpret(LivySparkInterpreter.java:101) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:341) at org.apache.zeppelin.scheduler.Job.run(Job.java:176) at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.springframework.web.client.HttpClientErrorException: 401 Authentication required at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91) at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:667) at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:620) at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecuteSubject(KerberosRestTemplate.java:202) at org.springframework.security.kerberos.client.KerberosRestTemplate.access$100(KerberosRestTemplate.java:67) at org.springframework.security.kerberos.client.KerberosRestTemplate$1.run(KerberosRestTemplate.java:191) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:187)

I have set the below in the Livy interpreter:

zeppelin.livy.keytab = /home/zeppelin/zeppelin.keytab

zeppelin.livy.principal = zeppelin/host.com@DOMAIN.COM

livy.spark.master = yarn-cluster (tried with blank as well)

Zeppelin itself has authentication enabled (Shiro with Active Directory) which is working fine and lets me login using my Active Directory credentials. Kerberized Livy also works fine with CURL when using 'negotiate' after kinit. When I kinit using the zeppelin keytab (zeppelin/host.com@DOMAIN.COM), it works fine as well (rules out keytab issues) and I am able to succesfully submit CURL negotiate GET & PUT requsts to Livy. Within Livy, zeppelin is set as a super user. Any pointers would be much appreciated.

[#2]

@Prajwal Kumar

Can you please verify if you have hadoop proxyuser set for livy in core-site.xml.

And also the error show "Server not found in Kerberos database" and the principal used is "zeppelin/host.com@DOMAIN.COM" which doesnt seems to be correct. /home/zeppelin/zeppelin.keytab keytab mentioned in interpreter must me a user principal. Something like "zeppelin-hdp25@RAGHAV.COM".

Create keytab with a user principal and use that in your interpreter. And also make sure below properties are set

<property>
  <name>hadoop.proxyuser.livy.groups</name>
  <value>*</value>
</property>
<property>
  <name>hadoop.proxyuser.livy.hosts</name>
  <value>*</value>
</property>

[#3]

@rguruvannagari

Thanks for the hint. Livy proxy user was already set. I have updated my Livy principal to xyzuser@DOMAIN.COM with the xyzuser being set as a super user in Livy. I have also successfully tested submission of jobs to Livy with cURL. But I still get an error in Zeppelin (this time error 400). Any pointers?

INFO [2017-01-09 01:08:29,805] ({pool-2-thread-4} SchedulerFactory.java[jobStarted]:131) - Job remoteInterpretJob_1483920509805 started by scheduler org.apache.zeppelin.livy.LivySparkInterpreter279557022 ERROR [2017-01-09 01:08:29,981] ({pool-2-thread-4} LivyHelper.java[createSession]:121) - Error getting session for user org.springframework.web.client.RestClientException: Error running rest call; nested exception is org.springframework.web.client.HttpClientErrorException: 400 Bad Request at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:196) at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:580) at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:498) at org.apache.zeppelin.livy.LivyHelper.executeHTTP(LivyHelper.java:344) at org.apache.zeppelin.livy.LivyHelper.createSession(LivyHelper.java:73) at org.apache.zeppelin.livy.LivySparkInterpreter.interpret(LivySparkInterpreter.java:101) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:341) at org.apache.zeppelin.scheduler.Job.run(Job.java:176) at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.springframework.web.client.HttpClientErrorException: 400 Bad Request at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91) at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:667) at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:620) at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecuteSubject(KerberosRestTemplate.java:202) at org.springframework.security.kerberos.client.KerberosRestTemplate.access$100(KerberosRestTemplate.java:67) at org.springframework.security.kerberos.client.KerberosRestTemplate$1.run(KerberosRestTemplate.java:191) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:187) ... 16 more ERROR [2017-01-09 01:08:29,983] ({pool-2-thread-4} LivySparkInterpreter.java[interpret]:108) - Exception in LivySparkInterpreter while interpret org.springframework.web.client.RestClientException: Error running rest call; nested exception is org.springframework.web.client.HttpClientErrorException: 400 Bad Request at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:196) at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:580) at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:498) at org.apache.zeppelin.livy.LivyHelper.executeHTTP(LivyHelper.java:344) at org.apache.zeppelin.livy.LivyHelper.createSession(LivyHelper.java:73) at org.apache.zeppelin.livy.LivySparkInterpreter.interpret(LivySparkInterpreter.java:101) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:341) at org.apache.zeppelin.scheduler.Job.run(Job.java:176) at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.springframework.web.client.HttpClientErrorException: 400 Bad Request at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91) at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:667) at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:620) at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecuteSubject(KerberosRestTemplate.java:202) at org.springframework.security.kerberos.client.KerberosRestTemplate.access$100(KerberosRestTemplate.java:67) at org.springframework.security.kerberos.client.KerberosRestTemplate$1.run(KerberosRestTemplate.java:191) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:187) ... 16 more

[#4 · Accepted solution]

Fixed this by leaving livy.spark.master blank (instead of yarn-cluster) as this value is black-listed in Livy.

[#5]

@Prajwal Kumar Did not work for me.

Removing it started giving new errors :

ERROR [2017-01-24 15:10:47,037] ({pool-2-thread-2} LivyHelper.java[createSession]:128) - Error getting session for user java.lang.Exception: Cannot start spark. at org.apache.zeppelin.livy.LivyHelper.createSession(LivyHelper.java:117) at org.apache.zeppelin.livy.LivySparkInterpreter.interpret(LivySparkInterpreter.java:101)

[#6]

I believe that the problem is that localhost is not a valid host for kerberos. I changed my livy interpreter settings to point out the actual host.

[#7]

Can you try livy.spark.master as YARN