Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

ClassNotFoundException Spark2-submit

Highlighted

ClassNotFoundException Spark2-submit

Explorer

Hi,

I'm getting this error below while trying to submit a scala jar built in Intellij using maven

 

  • spark version 2.3.0
  • scala version 2.11.11

 

Command used:

spark2-submit --master="yarn" --deploy-mode="cluster" --queue root.myyarnqueue --executor-memory 12G --driver-memory 12G --class MyClassName /projects/myscala.jar arg_1 arg_2

 

Error Message

java.lang.ClassNotFoundException: MyClassName
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.spark.util.Utils$.classForName(Utils.scala:239)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:851)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:198)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:228)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
20/01/27 15:54:47 INFO util.ShutdownHookManager: Shutdown hook called

2 REPLIES 2
Highlighted

Re: ClassNotFoundException Spark2-submit

Super Mentor

@ChineduLB 

 

The "MyClassName" class name is Fully Qualified Classname?  (It means do you have any Package name for this class? If yes then can you please specify the fully qualified classname like     "--class aaa.bbb.MyClassName")

 

Also can you please list the JAR to see if it has the class placed inside ti correctly? Please share the output of the mentioned command.

# jar -tvf  /projects/myscala.jar

# javap  -cp /projects/myscala.jar  MyClassName

.

 

Re: ClassNotFoundException Spark2-submit

Super Mentor

@ChineduLB 

Are you able to get this working?

Don't have an account?
Coming from Hortonworks? Activate your account here