HUE Spark notebook ClassNotFoundException

[#1]

Hi to all!

Do you know why I am keep getting this error below when running SparkWordCount jar app in HUE Spark Notebook? My HUE version is 3.9.0 (CDH 5.5.1). Do you think you can please help me?

Path: /user/hdfs/SparkWordCount-1.0-SNAPSHOT.jar
Class: cz.tmobile.surovecv.sparkwordcount.JavaWordCount

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/pkg/moip/data2_mzlp/mzpl/work/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/avro-tools-1.7.6-cdh5.5.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/pkg/moip/data2_mzlp/mzpl/work/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Warning: Local jar /user/hdfs/SparkWordCount-1.0-SNAPSHOT.jar does not exist, skipping.
java.lang.ClassNotFoundException: cz.tmobile.surovecv.sparkwordcount.JavaWordCount
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at org.apache.spark.util.Utils$.classForName(Utils.scala:173)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:639)
at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:162)
at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:160)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:160)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

[#2]

Are you running Livy in the default local mode? (not YARN)
What did you put in the jar input field?

[#3]

Hi Romain!

I am running like this: ./build/env/bin/hue livy_server as it was in the manual on gethue.com

...and what do you mean by the second question?

[#4]

Hi!
I am running in local mode and I did not put anything in the jar input field...

Thank you for any help!

[#5]

Here is the log from the command line when building the livy server:

$ ./build/env/bin/hue livy_server
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/pkg/moip/data2_mzlp/mzpl/work/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/livy-assembly.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/pkg/moip/data2_mzlp/mzpl/work/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
07:02:35.621 [main] INFO com.cloudera.hue.livy.server.Main$ - Using spark-submit version 1.5.0-cdh5.5.1
07:02:35.694 [main] INFO com.cloudera.hue.jetty.util.log - Logging initialized @6518ms
07:02:35.929 [main] INFO com.cloudera.hue.jetty.server.Server - jetty-9.2.z-SNAPSHOT
07:02:36.012 [main] INFO o.scalatra.servlet.ScalatraListener - The cycle class name from the config: com.cloudera.hue.livy.server.ScalatraBootstrap
07:02:36.038 [main] INFO o.scalatra.servlet.ScalatraListener - Initializing life cycle class: ScalatraBootstrap
07:02:36.044 [main] INFO c.c.h.livy.server.ScalatraBootstrap - Using Process() sessions
07:02:36.705 [main] INFO c.c.h.j.s.handler.ContextHandler - Started c.c.h.j.s.ServletContextHandler@7766c836{/,file:/pkg/moip/data2_mzlp/mzpl/work/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/src/main/com/cloudera/hue/livy/server,AVAILABLE}
07:02:36,705 |-INFO in null - Will use configuration resource [/logback-access.xml]
07:02:36,707 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@ed2be35 - URL [jar:file:/pkg/moip/data2_mzlp/mzpl/work/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/livy-assembly.jar!/logback-access.xml] is not of type file
07:02:36,714 |-INFO in ch.qos.logback.access.joran.action.ConfigurationAction - debug attribute not set
07:02:36,714 |-INFO in ch.qos.logback.core.joran.action.StatusListenerAction - Added status listener of type [ch.qos.logback.core.status.OnConsoleStatusListener]
07:02:36,715 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
07:02:36,715 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
07:02:36,716 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.access.PatternLayoutEncoder] for [encoder] property
07:02:36,735 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to null
07:02:36,735 |-INFO in ch.qos.logback.access.joran.action.ConfigurationAction - End of configuration.
07:02:36,736 |-INFO in ch.qos.logback.access.joran.JoranConfigurator@60d2b97b - Registering current configuration as safe fallback point
07:02:36,705 |-INFO in LogbackRequestLog - Will use configuration resource [/logback-access.xml]
07:02:36,707 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@ed2be35 - URL [jar:file:/pkg/moip/data2_mzlp/mzpl/work/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/livy-assembly.jar!/logback-access.xml] is not of type file
07:02:36,714 |-INFO in ch.qos.logback.access.joran.action.ConfigurationAction - debug attribute not set
07:02:36,714 |-INFO in ch.qos.logback.core.joran.action.StatusListenerAction - Added status listener of type [ch.qos.logback.core.status.OnConsoleStatusListener]
07:02:36,715 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
07:02:36,715 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
07:02:36,716 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.access.PatternLayoutEncoder] for [encoder] property
07:02:36,735 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to null
07:02:36,735 |-INFO in ch.qos.logback.access.joran.action.ConfigurationAction - End of configuration.
07:02:36,736 |-INFO in ch.qos.logback.access.joran.JoranConfigurator@60d2b97b - Registering current configuration as safe fallback point

07:02:36.757 [main] INFO c.c.hue.jetty.server.ServerConnector - Started ServerConnector@3e3d6c70{HTTP/1.1}{localhost:8998}
07:02:40.954 [main] INFO com.cloudera.hue.jetty.server.Server - Started @11779ms
07:02:40.959 [main] INFO com.cloudera.hue.livy.WebServer - Starting server on 8998
07:07:05.436 [ForkJoinPool-1-worker-1] INFO c.c.h.l.s.SparkSubmitProcessBuilder - Running /pkg/moip/data2_mzlp/mzpl/work/cloudera/parcels/CDH/lib/spark/bin/spark-submit --name Livy --class cz.tmobile.surovecv.sparkwordcount.JavaWordCount --proxy-user hdfs file:////user/cdrstore/jars/SparkWordCount-1.0-SNAPSHOT.jar /user/cdrstore/pokus.txt 2
07:07:05.460 [ForkJoinPool-1-worker-1] INFO c.c.hue.livy.server.SessionManager - created session 0

[#6 · Accepted solution]

Hello,

So I solved by uploading the jar file on the (local) unix filesystem (with prefix file://) instead of using it from HDFS. I don't know why, but even with the prefix "hdfs://" HUE or Spark was not able to use it from that Path.

Václav

[#7]

Hi Guys,

is there is an  alternative to the --jars option of spark-submit in the spark notebook in Hue?