Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Livy: object Predef does not have a member classOf

avatar
New Contributor

Hello,

 

I'm trying to import a package with Livy but it keeps hitting this error when I create the session:

 

scala.reflect.internal.FatalError: object Predef does not have a member classOf

This are the libraries that i tried to import:

 

 

livy.spark.jars.packages     com.ubirch.key:model-db_2.11:0.4.0
                             org.scala-lang:scala-parser-combinators:2.11.0-M4

 

So i tried to exclude the reflect library using this parameter:

 

spark.jars.excludes    org.scala-lang:scala-reflect:2.11.8

 

But this is not working. Also when I import those libraries directly into spark with --packages it works fine.

 

I think i'm hitting this bug.

 

My Livy version is livy-0.5.0-incubating-bin.

 

Regards,

Silva

2 REPLIES 2

avatar
New Contributor

I faced the same issue till now. When i'm adding my uber jar to livy-rsc-jars I was getting this error. I was able to reproduce this when i placed the jar in spark-client/jars and started spark-shell (spark shell stops with an exception class not found). The problem was i built my uber jar using java-9 in my local and in my spark cluster it is java-8 once i built using java-8 this got resolved. 

 

Regards,

Satish Tvv.

avatar
New Contributor

apparently this happend again for another project. And the issue was my assembly jar was bringing scala-library classes as well. So in my build.sbt i have excluded the scala by the following and that solved the issue.

assemblyOption in assembly ~= { _.copy(includeScala = false) }

 After this change assembly jar didn't have scala packages, and livy started working