Reply
Highlighted
LL
New Contributor
Posts: 1
Registered: ‎09-12-2018

livy 0.5, spark 2.3 and kyro

Hi Folks,

 

I've been experimenting with livy and have been hitting a snag with Kyro and the first job submission. Has anyone encountered the attached stack trace when trying run the pyspark livy example?

 

 

I've been using (0.5 - 0.6) from here (as there are references to spark2.3 being supported in the source):

 

https://github.com/apache/incubator-livy

 

as opposed to this repo, which looks to be untouched for some time now:

 

https://github.com/cloudera/livy

 

 

 

I've found at least one other person hitting the same issue here:

 

https://jira.apache.org/jira/browse/LIVY-459

 

 

 

 

Does anyone have any workarounds (other than resubmitting the same job) ?

 

Thanks in advance!

 

 

 

 

 

*** Exception: org.apache.livy.shaded.kryo.kryo.KryoException: Encountered unregistered class ID: 510                                   
org.apache.livy.shaded.kryo.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119)                                     
org.apache.livy.shaded.kryo.kryo.Kryo.readClass(Kryo.java:656)      
org.apache.livy.shaded.kryo.kryo.Kryo.readClassAndObject(Kryo.java:767)                                                                 
org.apache.livy.client.common.Serializer.deserialize(Serializer.java:63)                                                                
org.apache.livy.rsc.driver.BypassJob.call(BypassJob.java:39)        
org.apache.livy.rsc.driver.BypassJob.call(BypassJob.java:27)        
org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:57)      
org.apache.livy.rsc.driver.BypassJobWrapper.call(BypassJobWrapper.java:42)                                                              
org.apache.livy.rsc.driver.BypassJobWrapper.call(BypassJobWrapper.java:27)                                                              
java.util.concurrent.FutureTask.run(FutureTask.java:266)            
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)                                                         
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)                                                         
java.lang.Thread.run(Thread.java:745)

Announcements