Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

livy 0.5, spark 2.3 and kyro

livy 0.5, spark 2.3 and kyro

New Contributor

Hi Folks,

 

I've been experimenting with livy and have been hitting a snag with Kyro and the first job submission. Has anyone encountered the attached stack trace when trying run the pyspark livy example?

 

 

I've been using (0.5 - 0.6) from here (as there are references to spark2.3 being supported in the source):

 

https://github.com/apache/incubator-livy

 

as opposed to this repo, which looks to be untouched for some time now:

 

https://github.com/cloudera/livy

 

 

 

I've found at least one other person hitting the same issue here:

 

https://jira.apache.org/jira/browse/LIVY-459

 

 

 

 

Does anyone have any workarounds (other than resubmitting the same job) ?

 

Thanks in advance!

 

 

 

 

 

*** Exception: org.apache.livy.shaded.kryo.kryo.KryoException: Encountered unregistered class ID: 510                                   
org.apache.livy.shaded.kryo.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119)                                     
org.apache.livy.shaded.kryo.kryo.Kryo.readClass(Kryo.java:656)      
org.apache.livy.shaded.kryo.kryo.Kryo.readClassAndObject(Kryo.java:767)                                                                 
org.apache.livy.client.common.Serializer.deserialize(Serializer.java:63)                                                                
org.apache.livy.rsc.driver.BypassJob.call(BypassJob.java:39)        
org.apache.livy.rsc.driver.BypassJob.call(BypassJob.java:27)        
org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:57)      
org.apache.livy.rsc.driver.BypassJobWrapper.call(BypassJobWrapper.java:42)                                                              
org.apache.livy.rsc.driver.BypassJobWrapper.call(BypassJobWrapper.java:27)                                                              
java.util.concurrent.FutureTask.run(FutureTask.java:266)            
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)                                                         
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)                                                         
java.lang.Thread.run(Thread.java:745)