Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

HUE Spark notebook ClassNotFoundException

avatar
Rising Star

Hi to all!

Do you know why I am keep getting this error below when running SparkWordCount jar app in HUE Spark Notebook? My HUE version is 3.9.0 (CDH 5.5.1). Do you think you can please help me?

 

Path: /user/hdfs/SparkWordCount-1.0-SNAPSHOT.jar
Class: cz.tmobile.surovecv.sparkwordcount.JavaWordCount

 

error.PNG

 

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/pkg/moip/data2_mzlp/mzpl/work/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/avro-tools-1.7.6-cdh5.5.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/pkg/moip/data2_mzlp/mzpl/work/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Warning: Local jar /user/hdfs/SparkWordCount-1.0-SNAPSHOT.jar does not exist, skipping.
java.lang.ClassNotFoundException: cz.tmobile.surovecv.sparkwordcount.JavaWordCount
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at org.apache.spark.util.Utils$.classForName(Utils.scala:173)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:639)
at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:162)
at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:160)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:160)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

1 ACCEPTED SOLUTION

avatar
Rising Star

Hello,

 

So I solved by uploading the jar file on the (local) unix filesystem (with prefix file://) instead of using it from HDFS. I don't know why, but even with the prefix "hdfs://" HUE or Spark was not able to use it from that Path.

 

Václav

View solution in original post

6 REPLIES 6

avatar
Super Guru
Are you running Livy in the default local mode? (not YARN)
What did you put in the jar input field?

avatar
Rising Star

Hi Romain!

 

I am running like this: ./build/env/bin/hue livy_server as it was in the manual on gethue.com

 

...and what do you mean by the second question?

avatar
Rising Star

Hi!
I am running in local mode and I did not put anything in the jar input field...

Thank you for any help!

avatar
Rising Star

Here is the log from the command line when building the livy server:

 

$ ./build/env/bin/hue livy_server
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/pkg/moip/data2_mzlp/mzpl/work/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/livy-assembly.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/pkg/moip/data2_mzlp/mzpl/work/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
07:02:35.621 [main] INFO com.cloudera.hue.livy.server.Main$ - Using spark-submit version 1.5.0-cdh5.5.1
07:02:35.694 [main] INFO com.cloudera.hue.jetty.util.log - Logging initialized @6518ms
07:02:35.929 [main] INFO com.cloudera.hue.jetty.server.Server - jetty-9.2.z-SNAPSHOT
07:02:36.012 [main] INFO o.scalatra.servlet.ScalatraListener - The cycle class name from the config: com.cloudera.hue.livy.server.ScalatraBootstrap
07:02:36.038 [main] INFO o.scalatra.servlet.ScalatraListener - Initializing life cycle class: ScalatraBootstrap
07:02:36.044 [main] INFO c.c.h.livy.server.ScalatraBootstrap - Using Process() sessions
07:02:36.705 [main] INFO c.c.h.j.s.handler.ContextHandler - Started c.c.h.j.s.ServletContextHandler@7766c836{/,file:/pkg/moip/data2_mzlp/mzpl/work/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/src/main/com/cloudera/hue/livy/server,AVAILABLE}
07:02:36,705 |-INFO in null - Will use configuration resource [/logback-access.xml]
07:02:36,707 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@ed2be35 - URL [jar:file:/pkg/moip/data2_mzlp/mzpl/work/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/livy-assembly.jar!/logback-access.xml] is not of type file
07:02:36,714 |-INFO in ch.qos.logback.access.joran.action.ConfigurationAction - debug attribute not set
07:02:36,714 |-INFO in ch.qos.logback.core.joran.action.StatusListenerAction - Added status listener of type [ch.qos.logback.core.status.OnConsoleStatusListener]
07:02:36,715 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
07:02:36,715 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
07:02:36,716 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.access.PatternLayoutEncoder] for [encoder] property
07:02:36,735 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to null
07:02:36,735 |-INFO in ch.qos.logback.access.joran.action.ConfigurationAction - End of configuration.
07:02:36,736 |-INFO in ch.qos.logback.access.joran.JoranConfigurator@60d2b97b - Registering current configuration as safe fallback point
07:02:36,705 |-INFO in LogbackRequestLog - Will use configuration resource [/logback-access.xml]
07:02:36,707 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@ed2be35 - URL [jar:file:/pkg/moip/data2_mzlp/mzpl/work/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/livy-assembly.jar!/logback-access.xml] is not of type file
07:02:36,714 |-INFO in ch.qos.logback.access.joran.action.ConfigurationAction - debug attribute not set
07:02:36,714 |-INFO in ch.qos.logback.core.joran.action.StatusListenerAction - Added status listener of type [ch.qos.logback.core.status.OnConsoleStatusListener]
07:02:36,715 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
07:02:36,715 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
07:02:36,716 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.access.PatternLayoutEncoder] for [encoder] property
07:02:36,735 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to null
07:02:36,735 |-INFO in ch.qos.logback.access.joran.action.ConfigurationAction - End of configuration.
07:02:36,736 |-INFO in ch.qos.logback.access.joran.JoranConfigurator@60d2b97b - Registering current configuration as safe fallback point

07:02:36.757 [main] INFO c.c.hue.jetty.server.ServerConnector - Started ServerConnector@3e3d6c70{HTTP/1.1}{localhost:8998}
07:02:40.954 [main] INFO com.cloudera.hue.jetty.server.Server - Started @11779ms
07:02:40.959 [main] INFO com.cloudera.hue.livy.WebServer - Starting server on 8998
07:07:05.436 [ForkJoinPool-1-worker-1] INFO c.c.h.l.s.SparkSubmitProcessBuilder - Running /pkg/moip/data2_mzlp/mzpl/work/cloudera/parcels/CDH/lib/spark/bin/spark-submit --name Livy --class cz.tmobile.surovecv.sparkwordcount.JavaWordCount --proxy-user hdfs file:////user/cdrstore/jars/SparkWordCount-1.0-SNAPSHOT.jar /user/cdrstore/pokus.txt 2
07:07:05.460 [ForkJoinPool-1-worker-1] INFO c.c.hue.livy.server.SessionManager - created session 0

avatar
Rising Star

Hello,

 

So I solved by uploading the jar file on the (local) unix filesystem (with prefix file://) instead of using it from HDFS. I don't know why, but even with the prefix "hdfs://" HUE or Spark was not able to use it from that Path.

 

Václav

avatar
Master Collaborator

Hi Guys,

 

is there is an  alternative to the --jars option of spark-submit in the spark notebook in Hue?