Member since
01-06-2017
3
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 4806 | 01-09-2017 12:34 AM |
01-09-2017
12:34 AM
Fixed this by leaving livy.spark.master blank (instead of yarn-cluster) as this value is black-listed in Livy.
... View more
01-09-2017
12:19 AM
@rguruvannagari
Thanks for the hint. Livy proxy user was already set. I have updated my Livy principal to xyzuser@DOMAIN.COM with the xyzuser being set as a super user in Livy. I have also successfully tested submission of jobs to Livy with cURL. But I still get an error in Zeppelin (this time error 400). Any pointers? INFO [2017-01-09 01:08:29,805] ({pool-2-thread-4} SchedulerFactory.java[jobStarted]:131) - Job remoteInterpretJob_1483920509805 started by scheduler org.apache.zeppelin.livy.LivySparkInterpreter279557022 ERROR [2017-01-09 01:08:29,981] ({pool-2-thread-4} LivyHelper.java[createSession]:121) - Error getting session for user org.springframework.web.client.RestClientException: Error running rest call; nested exception is org.springframework.web.client.HttpClientErrorException: 400 Bad Request at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:196) at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:580) at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:498) at org.apache.zeppelin.livy.LivyHelper.executeHTTP(LivyHelper.java:344) at org.apache.zeppelin.livy.LivyHelper.createSession(LivyHelper.java:73) at org.apache.zeppelin.livy.LivySparkInterpreter.interpret(LivySparkInterpreter.java:101) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:341) at org.apache.zeppelin.scheduler.Job.run(Job.java:176) at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.springframework.web.client.HttpClientErrorException: 400 Bad Request at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91) at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:667) at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:620) at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecuteSubject(KerberosRestTemplate.java:202) at org.springframework.security.kerberos.client.KerberosRestTemplate.access$100(KerberosRestTemplate.java:67) at org.springframework.security.kerberos.client.KerberosRestTemplate$1.run(KerberosRestTemplate.java:191) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:187) ... 16 more ERROR [2017-01-09 01:08:29,983] ({pool-2-thread-4} LivySparkInterpreter.java[interpret]:108) - Exception in LivySparkInterpreter while interpret org.springframework.web.client.RestClientException: Error running rest call; nested exception is org.springframework.web.client.HttpClientErrorException: 400 Bad Request at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:196) at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:580) at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:498) at org.apache.zeppelin.livy.LivyHelper.executeHTTP(LivyHelper.java:344) at org.apache.zeppelin.livy.LivyHelper.createSession(LivyHelper.java:73) at org.apache.zeppelin.livy.LivySparkInterpreter.interpret(LivySparkInterpreter.java:101) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:341) at org.apache.zeppelin.scheduler.Job.run(Job.java:176) at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.springframework.web.client.HttpClientErrorException: 400 Bad Request at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91) at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:667) at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:620) at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecuteSubject(KerberosRestTemplate.java:202) at org.springframework.security.kerberos.client.KerberosRestTemplate.access$100(KerberosRestTemplate.java:67) at org.springframework.security.kerberos.client.KerberosRestTemplate$1.run(KerberosRestTemplate.java:191) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:187) ... 16 more
... View more
01-07-2017
08:14 PM
1 Kudo
I have Zeppelin 0.6.2 pre-built binary with Kerberised Livy 0.2 on a Kerberized Hadoop/Spark cluster. When I use the Livy interpreter with Spark, I hit the below error in the zeppelin logs (the livy logs do not show anything as the connection is not hitting Livy due to a failure in zeppelin) INFO [2017-01-06 12:53:05,121] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:131) - Job remoteInterpretJob_1483707185119 started by scheduler org.apache.zeppelin.livy.LivySparkInterpreter1764226124
WARN [2017-01-06 12:53:05,891] ({pool-2-thread-2} HttpAuthenticator.java[generateAuthResponse]:207) - NEGOTIATE authentication error: No valid credentials provided (Mechanism level: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7)))
ERROR [2017-01-06 12:53:05,917] ({pool-2-thread-2} LivyHelper.java[createSession]:121) - Error getting session for user
org.springframework.web.client.RestClientException: Error running rest call; nested exception is org.springframework.web.client.HttpClientErrorException: 401 Authentication required
at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:196)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:580)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:498)
at org.apache.zeppelin.livy.LivyHelper.executeHTTP(LivyHelper.java:344)
at org.apache.zeppelin.livy.LivyHelper.createSession(LivyHelper.java:73)
at org.apache.zeppelin.livy.LivySparkInterpreter.interpret(LivySparkInterpreter.java:101)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:341)
at org.apache.zeppelin.scheduler.Job.run(Job.java:176)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.springframework.web.client.HttpClientErrorException: 401 Authentication required
at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91)
at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:667)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:620)
at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecuteSubject(KerberosRestTemplate.java:202)
at org.springframework.security.kerberos.client.KerberosRestTemplate.access$100(KerberosRestTemplate.java:67)
at org.springframework.security.kerberos.client.KerberosRestTemplate$1.run(KerberosRestTemplate.java:191)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at org.springframework.security.kerberos.client.KerberosRestTemplate.doExecute(KerberosRestTemplate.java:187) I have set the below in the Livy interpreter: zeppelin.livy.keytab = /home/zeppelin/zeppelin.keytab zeppelin.livy.principal = zeppelin/host.com@DOMAIN.COM livy.spark.master = yarn-cluster (tried with blank as well) Zeppelin itself has authentication enabled (Shiro with Active Directory) which is working fine and lets me login using my Active Directory credentials. Kerberized Livy also works fine with CURL when using 'negotiate' after kinit. When I kinit using the zeppelin keytab (zeppelin/host.com@DOMAIN.COM), it works fine as well (rules out keytab issues) and I am able to succesfully submit CURL negotiate GET & PUT requsts to Livy. Within Livy, zeppelin is set as a super user.
Any pointers would be much appreciated.
... View more
Labels:
- Labels:
-
Apache Zeppelin