Cloudera Community
jarededrake
user rank
Explorer

Re: Oozie work flow not finding main class

by Explorer in Support Questions
‎04-26-2022 09:14 AM
‎04-26-2022 09:14 AM
Hey @mszurap , no problem at all totally understand.  So I ran my "SparkTutorial-1.0-SNAPSHOT.jar" from Hue > Query > Scheduler > Workflow, I then drag down Spark to the work flow, I then fill out the inputs like this:  Jar/py name: SParkTutorial-10.0.SNAPSHOT.jar Main class: Main  Files + : /user/zzmdrakej2/SparkTutorial-1.0-SNAPSHOT.jar   I then get this error, its different so I guess that's a good sign right:    Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, Delegation Token can be issued only with kerberos or web authentication     ... View more

Re: Oozie work flow not finding main class

by Explorer in Support Questions
‎04-15-2022 08:31 AM
‎04-15-2022 08:31 AM
Forgot to add the contents of    jar tf C:\Users\drakej\Desktop\SparkTutorial\target\SparkTutorial-1.0-SNAPSHOT.jar   META-INF/ META-INF/MANIFEST.MF Hortonworks/ Hortonworks/SparkTutorial/ code.txt Hortonworks/SparkTutorial/Main.class Main.class replacementValues.properties shakespeareText.txt ULAN-Test-IPSummary.csv ULAN-Test-IPSummary.txt META-INF/maven/ META-INF/maven/hortonworks/ META-INF/maven/hortonworks/SparkTutorial/ META-INF/maven/hortonworks/SparkTutorial/pom.xml META-INF/maven/hortonworks/SparkTutorial/pom.properties ... View more

Re: Oozie work flow not finding main class

by Explorer in Support Questions
‎04-15-2022 08:29 AM
‎04-15-2022 08:29 AM
So this is interesting, I run "mvn package" to package my application into a jar and I get two different jars I get a SparkTutorial.jar. When I look at the contents of that jar file I only see my dependencies but I do not see my main class    I run: jar tf C:\Users\drakej\Desktop\SparkTutorial\out\artifacts\SparkTutorial_jar\SparkTutorial.jar.   Sample of the output is below, it only lists my dependencies.    org/apache/hadoop/ha/proto/HAServiceProtocolProtos$TransitionToStandbyRequestProto$Builder.class org/apache/hadoop/ha/proto/HAServiceProtocolProtos$TransitionToStandbyRequestProto.class org/apache/hadoop/ha/proto/HAServiceProtocolProtos$TransitionToStandbyRequestProtoOrBuilder.class org/apache/hadoop/ha/proto/HAServiceProtocolProtos$TransitionToStandbyResponseProto$1.class org/apache/hadoop/ha/proto/HAServiceProtocolProtos$TransitionToStandbyResponseProto$Builder.class org/apache/hadoop/ha/proto/HAServiceProtocolProtos$TransitionToStandbyResponseProto.class org/apache/hadoop/ha/proto/HAServiceProtocolProtos$TransitionToStandbyResponseProtoOrBuilder.class org/apache/hadoop/ha/proto/HAServiceProtocolProtos.class org/apache/hadoop/ha/proto/ZKFCProtocolProtos$1.class org/apache/hadoop/ha/proto/ZKFCProtocolProtos$CedeActiveRequestProto$1.class org/apache/hadoop/ha/proto/ZKFCProtocolProtos$CedeActiveRequestProto$Builder.class org/apache/hadoop/ha/proto/ZKFCProtocolProtos$CedeActiveRequestProto.class org/apache/hadoop/ha/proto/ZKFCProtocolProtos$CedeActiveRequestProtoOrBuilder.class org/apache/hadoop/ha/proto/ZKFCProtocolProtos$CedeActiveResponseProto$1.class org/apache/hadoop/ha/proto/ZKFCProtocolProtos$CedeActiveResponseProto$Builder.class org/apache/hadoop/ha/proto/ZKFCProtocolProtos$CedeActiveResponseProto.class org/apache/hadoop/ha/proto/ZKFCProtocolProtos$CedeActiveResponseProtoOrBuilder.class org/apache/hadoop/ha/proto/ZKFCProtocolProtos$GracefulFailoverRequestProto$1.class org/apache/hadoop/ha/proto/ZKFCProtocolProtos$GracefulFailoverRequestProto$Builder.class   The second jar file that is made is SparkTutorial-1.0-SNAPSHOT.jar, I run    jar tf C:\Users\drakej\Desktop\SparkTutorial\target\SparkTutorial-1.0-SNAPSHOT.jar Then there it has my class listed, however when I run this jar file in oozie I get the error:   org.apache.oozie.action.hadoop.JavaMainException: java.lang.NoClassDefFoundError: org/apache/spark/SparkConf  So one jar has my dependencies and another jar has my class but they are not together.  ... View more

Re: Oozie work flow not finding main class

by Explorer in Support Questions
‎04-15-2022 08:11 AM
‎04-15-2022 08:11 AM
I can yes I will get those screen shots to you ... View more

Re: Oozie work flow not finding main class

by Explorer in Support Questions
‎04-14-2022 08:18 AM
‎04-14-2022 08:18 AM
@mszurap You were right I didn't have the "s" at the end 😐. However I am still getting the problem on oozie, I've add a screen shot so you can see the structure of my program and you can see in the MANIFEST.MF file it has my Main class as  Hortonworks.SparkTutorial.Main ... View more

Oozie work flow not finding main class

by Explorer in Support Questions
‎04-12-2022 08:47 AM
‎04-12-2022 08:47 AM
Hello Community,   I am going through this tutorial https://www.cloudera.com/tutorials/setting-up-a-spark-development-environment-with-java.html and I am trying to submit it through Oozie work flow. In Hue I am going to Query > Scheduler > Workflow then dropping Java program to actions and then uploading jar file then adding the main class which is Hortonworks.SparkTutorial.Main. When I click to run the Oozie work flow job I keep getting the error of: "Caused by: java.lang.ClassNotFoundException: Class Hortonwork.SparkTutorial.Main not found". I'm using intelliJ to do this project so I hold down ctrl and hover over Main and it takes me to MANIFEST.MF file and it says Main-Class: Hortonworks.SparkTutorial.Main so I'm getting the main class definition right I feel like. I cannot figure out why it is saying it can't find my class.     ... View more
Labels:

Re: Getting "Caused by: java.lang.NoClassDefFoundE...

by Explorer in Support Questions
‎04-11-2022 08:28 AM
‎04-11-2022 08:28 AM
Sorry for the late reply thank you for you help with this, I will try to run it with Oozie instead. So it is not recommended at all to run spark jobs created with Java from Hue > Query > Editor > Java?  ... View more

Getting "Caused by: java.lang.NoClassDefFoundError...

by Explorer in Support Questions
‎03-29-2022 07:36 AM
‎03-29-2022 07:36 AM
Hello Community, I'm running into a problem when I submit a job in Hue editor, from Hue home page I navigate to Query > Editor > Java. I point to the path of my jar file and then add the class name called Hortonwork.SparkTutorial.Main. Then after I hit run I get this error   "Caused by: java.lang.NoClassDefFoundError: org/apache/spark/SparkConf".   I imagine it has to with not having the jar file SparkConf, I'm using IntelliJ for this project and I've added some jars to my project, I've taken some screen shots of what the jar files look like and have attached them to the post. Curious if any of these jars look familiar to anyone, I may not even be adding the right ones.    Thank you all for the taking the time to read my post and for your help.    Full list error is:    >>> Invoking Main class now >>> INFO: loading log4j config file log4j.properties. INFO: log4j config file log4j.properties loaded successfully. Setting [tez.application.tags] tag: oozie-4908088a838f5d2717d8d8aabe5ccc23 Setting [spark.yarn.tags] tag: oozie-4908088a838f5d2717d8d8aabe5ccc23 Fetching child yarn jobs tag id : oozie-4908088a838f5d2717d8d8aabe5ccc23 Child yarn jobs are found - Main class : Hortonworks.SparkTutorial.Main Arguments : <<< Invocation of Main class completed <<< Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.JavaMain], main() threw exception, java.lang.NoClassDefFoundError: org/apache/spark/SparkConf org.apache.oozie.action.hadoop.JavaMainException: java.lang.NoClassDefFoundError: org/apache/spark/SparkConf at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:58) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:81) at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:35) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:235) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.NoClassDefFoundError: org/apache/spark/SparkConf at Hortonworks.SparkTutorial.Main.main(Main.java:16) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:55) ... 15 more Caused by: java.lang.ClassNotFoundException: org.apache.spark.SparkConf at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 21 more Oozie Launcher failed, finishing Hadoop job gracefully Oozie Launcher, uploading action data to HDFS sequence file: hdfs://HSV-ULANZoneB-nameservice1/user/zzmdrakej2/oozie-oozi/0000006-211115170934680-oozie-oozi-W/java-c507--java/action-data.seq Successfully reset security manager from org.apache.oozie.action.hadoop.LauncherSecurityManager@894858 to null Oozie Launcher ends   ... View more
Contact Me
Online
Offline
Last Visited
‎04-30-2022 01:11 PM
Community Statistics
Member Since ‎02-16-2022 07:55 AM
Last Visited ‎04-30-2022 01:11 PM
Posts 8