Reply
Highlighted
New Contributor
Posts: 3
Registered: ‎04-06-2018

Livy: object Predef does not have a member classOf

Hello,

 

I'm trying to import a package with Livy but it keeps hitting this error when I create the session:

 

scala.reflect.internal.FatalError: object Predef does not have a member classOf

This are the libraries that i tried to import:

 

 

livy.spark.jars.packages     com.ubirch.key:model-db_2.11:0.4.0
                             org.scala-lang:scala-parser-combinators:2.11.0-M4

 

So i tried to exclude the reflect library using this parameter:

 

spark.jars.excludes    org.scala-lang:scala-reflect:2.11.8

 

But this is not working. Also when I import those libraries directly into spark with --packages it works fine.

 

I think i'm hitting this bug.

 

My Livy version is livy-0.5.0-incubating-bin.

 

Regards,

Silva

New Contributor
Posts: 2
Registered: ‎06-06-2018

Re: Livy: object Predef does not have a member classOf

I faced the same issue till now. When i'm adding my uber jar to livy-rsc-jars I was getting this error. I was able to reproduce this when i placed the jar in spark-client/jars and started spark-shell (spark shell stops with an exception class not found). The problem was i built my uber jar using java-9 in my local and in my spark cluster it is java-8 once i built using java-8 this got resolved. 

 

Regards,

Satish Tvv.

New Contributor
Posts: 2
Registered: ‎06-06-2018

Re: Livy: object Predef does not have a member classOf

apparently this happend again for another project. And the issue was my assembly jar was bringing scala-library classes as well. So in my build.sbt i have excluded the scala by the following and that solved the issue.

assemblyOption in assembly ~= { _.copy(includeScala = false) }

 After this change assembly jar didn't have scala packages, and livy started working

Announcements