Created 04-06-2018 01:23 PM
Hello,
I'm trying to import a package with Livy but it keeps hitting this error when I create the session:
scala.reflect.internal.FatalError: object Predef does not have a member classOf
This are the libraries that i tried to import:
livy.spark.jars.packages com.ubirch.key:model-db_2.11:0.4.0 org.scala-lang:scala-parser-combinators:2.11.0-M4
So i tried to exclude the reflect library using this parameter:
spark.jars.excludes org.scala-lang:scala-reflect:2.11.8
But this is not working. Also when I import those libraries directly into spark with --packages it works fine.
I think i'm hitting this bug.
My Livy version is livy-0.5.0-incubating-bin.
Regards,
Silva
Created 06-06-2018 11:17 PM
I faced the same issue till now. When i'm adding my uber jar to livy-rsc-jars I was getting this error. I was able to reproduce this when i placed the jar in spark-client/jars and started spark-shell (spark shell stops with an exception class not found). The problem was i built my uber jar using java-9 in my local and in my spark cluster it is java-8 once i built using java-8 this got resolved.
Regards,
Satish Tvv.
Created 06-20-2018 10:25 AM
apparently this happend again for another project. And the issue was my assembly jar was bringing scala-library classes as well. So in my build.sbt i have excluded the scala by the following and that solved the issue.
assemblyOption in assembly ~= { _.copy(includeScala = false) }
After this change assembly jar didn't have scala packages, and livy started working