Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HUE Spark notebook ClassNotFoundException

Solved Go to solution

HUE Spark notebook ClassNotFoundException

Contributor

Hi to all!

Do you know why I am keep getting this error below when running SparkWordCount jar app in HUE Spark Notebook? My HUE version is 3.9.0 (CDH 5.5.1). Do you think you can please help me?

 

Path: /user/hdfs/SparkWordCount-1.0-SNAPSHOT.jar
Class: cz.tmobile.surovecv.sparkwordcount.JavaWordCount

 

error.PNG

 

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/pkg/moip/data2_mzlp/mzpl/work/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/avro-tools-1.7.6-cdh5.5.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/pkg/moip/data2_mzlp/mzpl/work/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Warning: Local jar /user/hdfs/SparkWordCount-1.0-SNAPSHOT.jar does not exist, skipping.
java.lang.ClassNotFoundException: cz.tmobile.surovecv.sparkwordcount.JavaWordCount
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:270)
at org.apache.spark.util.Utils$.classForName(Utils.scala:173)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:639)
at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:162)
at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:160)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:160)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

1 ACCEPTED SOLUTION

Accepted Solutions

Re: HUE Spark notebook ClassNotFoundException

Contributor

Hello,

 

So I solved by uploading the jar file on the (local) unix filesystem (with prefix file://) instead of using it from HDFS. I don't know why, but even with the prefix "hdfs://" HUE or Spark was not able to use it from that Path.

 

Václav

6 REPLIES 6

Re: HUE Spark notebook ClassNotFoundException

Are you running Livy in the default local mode? (not YARN)
What did you put in the jar input field?

Highlighted

Re: HUE Spark notebook ClassNotFoundException

Contributor

Hi Romain!

 

I am running like this: ./build/env/bin/hue livy_server as it was in the manual on gethue.com

 

...and what do you mean by the second question?

Re: HUE Spark notebook ClassNotFoundException

Contributor

Hi!
I am running in local mode and I did not put anything in the jar input field...

Thank you for any help!

Re: HUE Spark notebook ClassNotFoundException

Contributor

Here is the log from the command line when building the livy server:

 

$ ./build/env/bin/hue livy_server
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/pkg/moip/data2_mzlp/mzpl/work/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/livy-assembly.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/pkg/moip/data2_mzlp/mzpl/work/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
07:02:35.621 [main] INFO com.cloudera.hue.livy.server.Main$ - Using spark-submit version 1.5.0-cdh5.5.1
07:02:35.694 [main] INFO com.cloudera.hue.jetty.util.log - Logging initialized @6518ms
07:02:35.929 [main] INFO com.cloudera.hue.jetty.server.Server - jetty-9.2.z-SNAPSHOT
07:02:36.012 [main] INFO o.scalatra.servlet.ScalatraListener - The cycle class name from the config: com.cloudera.hue.livy.server.ScalatraBootstrap
07:02:36.038 [main] INFO o.scalatra.servlet.ScalatraListener - Initializing life cycle class: ScalatraBootstrap
07:02:36.044 [main] INFO c.c.h.livy.server.ScalatraBootstrap - Using Process() sessions
07:02:36.705 [main] INFO c.c.h.j.s.handler.ContextHandler - Started c.c.h.j.s.ServletContextHandler@7766c836{/,file:/pkg/moip/data2_mzlp/mzpl/work/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hue/src/main/com/cloudera/hue/livy/server,AVAILABLE}
07:02:36,705 |-INFO in null - Will use configuration resource [/logback-access.xml]
07:02:36,707 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@ed2be35 - URL [jar:file:/pkg/moip/data2_mzlp/mzpl/work/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/livy-assembly.jar!/logback-access.xml] is not of type file
07:02:36,714 |-INFO in ch.qos.logback.access.joran.action.ConfigurationAction - debug attribute not set
07:02:36,714 |-INFO in ch.qos.logback.core.joran.action.StatusListenerAction - Added status listener of type [ch.qos.logback.core.status.OnConsoleStatusListener]
07:02:36,715 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
07:02:36,715 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
07:02:36,716 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.access.PatternLayoutEncoder] for [encoder] property
07:02:36,735 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to null
07:02:36,735 |-INFO in ch.qos.logback.access.joran.action.ConfigurationAction - End of configuration.
07:02:36,736 |-INFO in ch.qos.logback.access.joran.JoranConfigurator@60d2b97b - Registering current configuration as safe fallback point
07:02:36,705 |-INFO in LogbackRequestLog - Will use configuration resource [/logback-access.xml]
07:02:36,707 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@ed2be35 - URL [jar:file:/pkg/moip/data2_mzlp/mzpl/work/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/livy-assembly.jar!/logback-access.xml] is not of type file
07:02:36,714 |-INFO in ch.qos.logback.access.joran.action.ConfigurationAction - debug attribute not set
07:02:36,714 |-INFO in ch.qos.logback.core.joran.action.StatusListenerAction - Added status listener of type [ch.qos.logback.core.status.OnConsoleStatusListener]
07:02:36,715 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
07:02:36,715 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
07:02:36,716 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.access.PatternLayoutEncoder] for [encoder] property
07:02:36,735 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to null
07:02:36,735 |-INFO in ch.qos.logback.access.joran.action.ConfigurationAction - End of configuration.
07:02:36,736 |-INFO in ch.qos.logback.access.joran.JoranConfigurator@60d2b97b - Registering current configuration as safe fallback point

07:02:36.757 [main] INFO c.c.hue.jetty.server.ServerConnector - Started ServerConnector@3e3d6c70{HTTP/1.1}{localhost:8998}
07:02:40.954 [main] INFO com.cloudera.hue.jetty.server.Server - Started @11779ms
07:02:40.959 [main] INFO com.cloudera.hue.livy.WebServer - Starting server on 8998
07:07:05.436 [ForkJoinPool-1-worker-1] INFO c.c.h.l.s.SparkSubmitProcessBuilder - Running /pkg/moip/data2_mzlp/mzpl/work/cloudera/parcels/CDH/lib/spark/bin/spark-submit --name Livy --class cz.tmobile.surovecv.sparkwordcount.JavaWordCount --proxy-user hdfs file:////user/cdrstore/jars/SparkWordCount-1.0-SNAPSHOT.jar /user/cdrstore/pokus.txt 2
07:07:05.460 [ForkJoinPool-1-worker-1] INFO c.c.hue.livy.server.SessionManager - created session 0

Re: HUE Spark notebook ClassNotFoundException

Contributor

Hello,

 

So I solved by uploading the jar file on the (local) unix filesystem (with prefix file://) instead of using it from HDFS. I don't know why, but even with the prefix "hdfs://" HUE or Spark was not able to use it from that Path.

 

Václav

Re: HUE Spark notebook ClassNotFoundException

Super Collaborator

Hi Guys,

 

is there is an  alternative to the --jars option of spark-submit in the spark notebook in Hue?