LivyServer exception

[#1]

Greetings Experts,

I am trying to run LivyServer in zeppelin in HDP 2.5TP sandbox, however my interpreter keeps failing. I tried %livy.sparkr and %livy.pyspark all getting same exceptions. See error messages below from the livy log file:

LivyPySparkInterpreter while interpret                  
java.lang.Exception: Cannot start  pyspark


INFO [2016-07-18 22:20:23,102] ({Thread-0} RemoteInterpreterServer.java[run]:81) - Starting remote interpreter server on port 41012                                 
 INFO [2016-07-18 22:20:23,420] ({pool-1-thread-4} RemoteInterpreterServer.java[createInterpreter]:169) - Instantiate interpreter org.apache.zeppelin.livy.LivySparkR
Interpreter                                                                                                                                                          
 INFO [2016-07-18 22:20:23,457] ({pool-1-thread-4} RemoteInterpreterServer.java[createInterpreter]:169) - Instantiate interpreter org.apache.zeppelin.livy.LivySparkI
nterpreter                                                                                                                                                           
 INFO [2016-07-18 22:20:23,461] ({pool-1-thread-4} RemoteInterpreterServer.java[createInterpreter]:169) - Instantiate interpreter org.apache.zeppelin.livy.LivyPySpar
kInterpreter                                                                                                                                                         
 INFO [2016-07-18 22:20:23,462] ({pool-1-thread-4} RemoteInterpreterServer.java[createInterpreter]:169) - Instantiate interpreter org.apache.zeppelin.livy.LivySparkS
QLInterpreter                                                                                                                                                        
 INFO [2016-07-18 22:20:23,505] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:131) - Job remoteInterpretJob_1468880423503 started by scheduler org.apache.zepp
elin.livy.LivySparkRInterpreter1573372211                                                                                                                            
ERROR [2016-07-18 22:20:24,722] ({pool-2-thread-2} LivyHelper.java[createSession]:89) - sessionId:0.0 state is starting                                              
ERROR [2016-07-18 22:20:25,754] ({pool-2-thread-2} LivyHelper.java[createSession]:89) - sessionId:0.0 state is starting                                              
ERROR [2016-07-18 22:20:26,769] ({pool-2-thread-2} LivyHelper.java[createSession]:89) - sessionId:0.0 state is starting                                              
ERROR [2016-07-18 22:20:27,784] ({pool-2-thread-2} LivyHelper.java[createSession]:89) - sessionId:0.0 state is dead                                                  
ERROR [2016-07-18 22:20:28,799] ({pool-2-thread-2} LivyHelper.java[createSession]:89) - sessionId:0.0 state is dead                                                  
ERROR [2016-07-18 22:20:29,814] ({pool-2-thread-2} LivyHelper.java[createSession]:89) - sessionId:0.0 state is dead   
....
....

ERROR [2016-07-18 22:41:10,602] ({pool-2-thread-6} LivyHelper.java[createSession]:89) - sessionId:2.0 state is starting                                              
ERROR [2016-07-18 22:41:11,609] ({pool-2-thread-6} LivyHelper.java[createSession]:114) - Error getting session for user within 60Sec.                                
ERROR [2016-07-18 22:41:11,609] ({pool-2-thread-6} LivyHelper.java[createSession]:120) - Error getting session for user                                              
java.lang.Exception: Cannot start  pyspark.                                                                                                                          
        at org.apache.zeppelin.livy.LivyHelper.createSession(LivyHelper.java:115)                                                                                    
        at org.apache.zeppelin.livy.LivyPySparkInterpreter.interpret(LivyPySparkInterpreter.java:63)                                                                 
        at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94)                                                                
        at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:341)                                      
        at org.apache.zeppelin.scheduler.Job.run(Job.java:176)                                                                                                       
        at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)                                                                                 
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)                                                                                   
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)                                                                                                  
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)                                     
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)                                            
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)                                                                           
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)                                                                           
        at java.lang.Thread.run(Thread.java:745)                                                                                                                     
ERROR [2016-07-18 22:41:11,610] ({pool-2-thread-6} LivyPySparkInterpreter.java[interpret]:70) - Exception in LivyPySparkInterpreter while interpret                  
java.lang.Exception: Cannot start  pyspark.                                                                                                                          
        at org.apache.zeppelin.livy.LivyHelper.createSession(LivyHelper.java:115)                                                                                    
        at org.apache.zeppelin.livy.LivyPySparkInterpreter.interpret(LivyPySparkInterpreter.java:63)                                                                 
        at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:94)                                                                
        at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:341)                                      
        at org.apache.zeppelin.scheduler.Job.run(Job.java:176)                                                                                                       
        at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)                                                                                 
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)                                                                                   
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)                                                                                                  
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)                                     
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)                                            
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)                                                                           
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)                                                                           
        at java.lang.Thread.run(Thread.java:745)                                                                                                                     
 INFO [2016-07-18 22:41:11,611] ({pool-2-thread-6} SchedulerFactory.java[jobFinished]:137) - Job remoteInterpretJob_1468881610078 finished by scheduler org.apache.ze
ppelin.livy.LivyPySparkInterpreter1321637285                

[#2]

@azeltov

Did you resolve your problem? I am facing the same.

[#3]

I am also facing this. @Edgar Daeds did you get it working?

Thanks.

[#4]

@azeltov , @Edgar Daeds

When we run anything from Livy server, it tries to connect to Resource manager at 8032 port. But in Sandbox, the value of yarn.resourcemanager.address is 8050 so it waits for 10 tries and then fails.

Go to Ambari->Yarn->Configs. Search for yarn.resourcemanager.address and change it to 8032. Restart YARN and then try running %livy.pyspark.

[#5]

@mrizvi

Actually on my cluster the value of yarn.resourcemanager.address is also 8050 (by default). I did not try your solution yet, but thank you for the answer. I am going to let you know if that works.

[#6]

I have seen this error for my customer. Issue was with memory footprint on the node hosting Zeppelin/Livy.

Free memory was 1GB. This was since livy had many dead session which were not releasing memory. Deleting livy sessions helped increasing free memory. YOu can use livy rest api to view sessions and delete dead sessions.

[#7]

Hi @azeltov

Can you restart the interpreter and give it a try again.

The session idles out leading to such errors.

Is your cluster kerberized?

[#8 · Accepted solution]

I ended up looking the livy.log file for the actual error message. My user that i was trying to run as did not have a home directory in HDFS. So created a home dir with proper permissions and got it working.