question Re: Zeppelin with kerberized Livy not working in Archives of Support Questions (Read Only)

Zeppelin with kerberized Livy not working

prajwal83 — Sun, 08 Jan 2017 04:14:15 GMT

I have Zeppelin 0.6.2 pre-built binary with Kerberised Livy 0.2 on a Kerberized Hadoop/Spark cluster. When I use the Livy interpreter with Spark, I hit the below error in the zeppelin logs (the livy logs do not show anything as the connection is not hitting Livy due to a failure in zeppelin)

INFO [2017-01-06 12:53:05,121] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:131) - Job remoteInterpretJob_1483707185119 started by scheduler org.apache.zeppelin.livy.LivySparkInterpreter1764226124 WARN [2017-01-06 12:53:05,891] ({pool-2-thread-2} HttpAuthenticator.java[generateAuthResponse]:207) - NEGOTIATE authentication error: No valid credentials provided (Mechanism level: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7))) ERROR [2017-01-06 12:53:05,917] ({pool-2-thread-2} LivyHelper.java[createSession]:121) - Error getting session for user org.springframework.web.client.RestClientException: Error running rest call; nested exception is org.springframework.web.client.HttpClientErrorException: 401 Authentication required at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:196) at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:580) at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:498) at org.apache.zeppelin.livy.LivyHelper.executeHTTP(LivyHelper.java:344) at org.apache.zeppelin.livy.LivyHelper.createSession(LivyHelper.java:73) at org.apache.zeppelin.livy.LivySparkInterpreter.interpret(LivySparkInterpreter.java:101) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:341) at org.apache.zeppelin.scheduler.Job.run(Job.java:176) at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.springframework.web.client.HttpClientErrorException: 401 Authentication required at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91) at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:667) at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:620) at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecuteSubject(KerberosRestTemplate.java:202) at org.springframework.security.kerberos.client.KerberosRestTemplate.access$100(KerberosRestTemplate.java:67) at org.springframework.security.kerberos.client.KerberosRestTemplate$1.run(KerberosRestTemplate.java:191) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:187)

I have set the below in the Livy interpreter:

zeppelin.livy.keytab = /home/zeppelin/zeppelin.keytab

zeppelin.livy.principal = zeppelin/host.com@DOMAIN.COM

livy.spark.master = yarn-cluster (tried with blank as well)

Zeppelin itself has authentication enabled (Shiro with Active Directory) which is working fine and lets me login using my Active Directory credentials. Kerberized Livy also works fine with CURL when using 'negotiate' after kinit. When I kinit using the zeppelin keytab (zeppelin/host.com@DOMAIN.COM), it works fine as well (rules out keytab issues) and I am able to succesfully submit CURL negotiate GET & PUT requsts to Livy. Within Livy, zeppelin is set as a super user. Any pointers would be much appreciated.

Re: Zeppelin with kerberized Livy not working

rguruvannagari — Sun, 08 Jan 2017 14:07:48 GMT

@Prajwal Kumar

Can you please verify if you have hadoop proxyuser set for livy in core-site.xml.

And also the error show "Server not found in Kerberos database" and the principal used is "zeppelin/host.com@DOMAIN.COM" which doesnt seems to be correct. /home/zeppelin/zeppelin.keytab keytab mentioned in interpreter must me a user principal. Something like "zeppelin-hdp25@RAGHAV.COM".

Create keytab with a user principal and use that in your interpreter. And also make sure below properties are set

<property>
  <name>hadoop.proxyuser.livy.groups</name>
  <value>*</value>
</property>
<property>
  <name>hadoop.proxyuser.livy.hosts</name>
  <value>*</value>
</property>

Re: Zeppelin with kerberized Livy not working

prajwal83 — Mon, 09 Jan 2017 08:19:55 GMT

@rguruvannagari

Thanks for the hint. Livy proxy user was already set. I have updated my Livy principal to xyzuser@DOMAIN.COM with the xyzuser being set as a super user in Livy. I have also successfully tested submission of jobs to Livy with cURL. But I still get an error in Zeppelin (this time error 400). Any pointers?

INFO [2017-01-09 01:08:29,805] ({pool-2-thread-4} SchedulerFactory.java[jobStarted]:131) - Job remoteInterpretJob_1483920509805 started by scheduler org.apache.zeppelin.livy.LivySparkInterpreter279557022 ERROR [2017-01-09 01:08:29,981] ({pool-2-thread-4} LivyHelper.java[createSession]:121) - Error getting session for user org.springframework.web.client.RestClientException: Error running rest call; nested exception is org.springframework.web.client.HttpClientErrorException: 400 Bad Request at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:196) at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:580) at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:498) at org.apache.zeppelin.livy.LivyHelper.executeHTTP(LivyHelper.java:344) at org.apache.zeppelin.livy.LivyHelper.createSession(LivyHelper.java:73) at org.apache.zeppelin.livy.LivySparkInterpreter.interpret(LivySparkInterpreter.java:101) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:341) at org.apache.zeppelin.scheduler.Job.run(Job.java:176) at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.springframework.web.client.HttpClientErrorException: 400 Bad Request at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91) at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:667) at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:620) at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecuteSubject(KerberosRestTemplate.java:202) at org.springframework.security.kerberos.client.KerberosRestTemplate.access$100(KerberosRestTemplate.java:67) at org.springframework.security.kerberos.client.KerberosRestTemplate$1.run(KerberosRestTemplate.java:191) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:187) ... 16 more ERROR [2017-01-09 01:08:29,983] ({pool-2-thread-4} LivySparkInterpreter.java[interpret]:108) - Exception in LivySparkInterpreter while interpret org.springframework.web.client.RestClientException: Error running rest call; nested exception is org.springframework.web.client.HttpClientErrorException: 400 Bad Request at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:196) at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:580) at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:498) at org.apache.zeppelin.livy.LivyHelper.executeHTTP(LivyHelper.java:344) at org.apache.zeppelin.livy.LivyHelper.createSession(LivyHelper.java:73) at org.apache.zeppelin.livy.LivySparkInterpreter.interpret(LivySparkInterpreter.java:101) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:341) at org.apache.zeppelin.scheduler.Job.run(Job.java:176) at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.springframework.web.client.HttpClientErrorException: 400 Bad Request at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91) at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:667) at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:620) at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecuteSubject(KerberosRestTemplate.java:202) at org.springframework.security.kerberos.client.KerberosRestTemplate.access$100(KerberosRestTemplate.java:67) at org.springframework.security.kerberos.client.KerberosRestTemplate$1.run(KerberosRestTemplate.java:191) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:187) ... 16 more

Re: Zeppelin with kerberized Livy not working

prajwal83 — Mon, 09 Jan 2017 08:34:03 GMT

Fixed this by leaving livy.spark.master blank (instead of yarn-cluster) as this value is black-listed in Livy.

Re: Zeppelin with kerberized Livy not working

smartninja723 — Tue, 24 Jan 2017 23:12:13 GMT

@Prajwal Kumar Did not work for me.

Removing it started giving new errors :

ERROR [2017-01-24 15:10:47,037] ({pool-2-thread-2} LivyHelper.java[createSession]:128) - Error getting session for user java.lang.Exception: Cannot start spark. at org.apache.zeppelin.livy.LivyHelper.createSession(LivyHelper.java:117) at org.apache.zeppelin.livy.LivySparkInterpreter.interpret(LivySparkInterpreter.java:101)

Re: Zeppelin with kerberized Livy not working

michael_salmon — Thu, 26 Jan 2017 15:56:15 GMT

I believe that the problem is that localhost is not a valid host for kerberos. I changed my livy interpreter settings to point out the actual host.

Re: Zeppelin with kerberized Livy not working

sameer_dalai — Sun, 05 May 2019 10:56:22 GMT

Can you try livy.spark.master as YARN