Getting error "User session not found 403" when using zeppelin with livy

[#1]

So we have a kerberized cluster and have integrated ldaps with zeppelin. But when i try to use livy i get the following error

ERROR [2016-12-01 10:00:02,442] ({pool-2-thread-4} LivyHelper.java[createSession]:128) - Error getting session for user java.lang.Exception: Cannot start spark. at org.apache.zeppelin.livy.LivyHelper.createSession(LivyHelper.java:117) at org.apache.zeppelin.livy.LivySparkSQLInterpreter.interpret(LivySparkSQLInterpreter.java:62) at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94) at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:341) at org.apache.zeppelin.scheduler.Job.run(Job.java:176) at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) ERROR [2016-12-01 10:00:02,442] ({pool-2-thread-4} LivySparkSQLInterpreter.java[interpret]:70) - Exception in LivySparkSQLInterpreter while interpret java.lang.Exception: Cannot start spark.

in zeppelin interpreter logs.

Also when i dont enable ldap for zeppelin (that is i log in as anonymous), I can see that job is been submitted to yarn but it fails as user is `zeppelin-clusterName`. But when i use ldap with zeppelin it is not even getting submit to yarn.

[#2]

Do you kerberos your cluster using ambari ? There're serveral configuration you need to make it work.

[#3]

Hey @jzhang, thanks for the interest. Yes, i have used ambari for kerberization and also i have set proxy users property for livy in hdfs

[#4]

The session might be expired, Can you restart livy interpreter ? And if you still get the error, check the RM UI to find the yarn app log.

[#5]

@jzhang, So i did restart the server and checked yarn logs but the job is not getting submitted, I cannot see any submitted or failed jobs.

[#6]

Do you run it in yarn-cluster mode ? Set livy.spark.master as yarn-cluster in interpreter setting page

[#7]

yes i have already set that to yarn-cluster mode

[#8]

@Bhavin Tandel Have you setup livy.superusers property to your zeppelin principal in livy.conf?

[#9]

yes, i have set it to zeppelin principal. But just for heads up i have not configured ldap for MIT Kerberos but i am using ldap with zeppelin.

[#10]

@Bhavin Tandel please ensure you have all of the following setup

1) when you try to login using ldap - the user you are getting logged in has corresponding data directory on HDFS (/user/xyz) ? Also, the unix user should be present on the cluster

2) livy.superusers = zeppelin-clustername in livy.conf (which you already have done)

3) Along with for 'livy' user, also please setup proxy permissions for both 'zeppelin' and 'zeppelin-clustername' in HDFS core-site.xml

hadoop.proxyuser.zeppelin.hosts = *
hadoop.proxyuser.zeppelin.hosts = *
hadoop.proxyuser.zeppelin-clustername.hosts = *
hadoop.proxyuser.zeppelin-clustername.hosts = *

[#11]

Hey @kbadani, thanks for getting back. I did this but no luck, I also checked after restarting the services but the 403 error persists.

[#12]

@Bhavin Tandel Can you please post what error you are getting in livy server logs ?

[#13]

@kbadani Sorry for the delay, I have attached the live.out log

livy-out.txt

[#14]

yarn app is started, could you check the yarn app log ? appid : application_1481021618182_0004

[#15 · Accepted solution]

So guys @kbadani and @jzhang the issue finally got solved and thank you for your support. It was very trivial but very important property, livy.superuser was set to `zeppelin-<cluster name in uppercase>` but the principal was in lower case 😛 . And changing that solved the issue, @kbadani You pointed out that property but I didnt know that it is case sensitive.

[#16]

@Bhavin Tandel good to know 🙂 Thanks