Reply
New Contributor
Posts: 1
Registered: ‎03-25-2016

Oozie Spark Action java.lang.NoClassDefFoundError: org/apache/spark/Logging

I'm starting my Spark app on CDH 5.7.4 with Kerberos using Oozie Spark Action and in yarn-cluster mode. I'm still getting an error:

Error: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/Logging
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.Logging
	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	... 13 more

What I cannot understand is that the same application started on front end node is working well. Moreover, I don't use the Logging class in my app.

 

Is it possible, that the spark shared lib in hdfs /user/oozie/share/lib/lib_<timestamp>/spark has a different state than the nominal SPARK_HOME=/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/lib/spark?

 

Here my workflow.xml file:

<workflow-app name="myApp workflow" xmlns="uri:oozie:workflow:0.5">
<global>
        <configuration>
            <property>
                <name>oozie.launcher.yarn.app.mapreduce.am.env</name>
                <value>SPARK_HOME=/opt/cloudera/parcels/CDH-5.7.4-1.cdh5.7.4.p0.2/lib/spark</value>
            </property>
        </configuration>
</global>
  <credentials>
    <credential name="hcat" type="hcat">
      <property>
        <name>hcat.metastore.uri</name>
        <value>thrift://<XXX></value>
      </property>
      <property>
        <name>hcat.metastore.principal</name>
        <value>hive/<XXX></value>
      </property>
    </credential>
  </credentials>
    <start to="spark"/>
    <kill name="Kill">
        <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <action name="spark" cred="hcat">
        <spark xmlns="uri:oozie:spark-action:0.1">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <master>yarn</master>
            <mode>cluster</mode>
            <name>AppName</name>
              <class>main.Class</class>
            <jar>${nameNode}/<path>/myJar.jar</jar>
              <spark-opts> --files ${nameNode}/<path>/hive-site.xml, ${nameNode}/<path>/file.conf</spark-opts>
              <arg>--mode history</arg>
              <arg>--base-timestamp 2016-01-20#00:00:00</arg>
              <arg>--conf file.conf</arg>
        </spark>
        <ok to="End"/>
        <error to="Kill"/>
    </action>
    <end name="End"/>
</workflow-app>

 Any thoughts on what I am missing?

Cloudera Employee
Posts: 35
Registered: ‎08-18-2014

Re: Oozie Spark Action java.lang.NoClassDefFoundError: org/apache/spark/Logging

@sarycz

Would you please try to use the Spark yarn-client mode instead of yarn-master mode? That is, change from

<mode>cluster</mode>

to 

<mode>client</mode>
New Contributor
Posts: 7
Registered: ‎06-14-2017

Re: Oozie Spark Action java.lang.NoClassDefFoundError: org/apache/spark/Logging

i use oozie with spark action by the master of yarn-client is sucess, but yarn-cluster is error;

can your leave your email,that i can contact you with tech question?

thank you very much.

New Contributor
Posts: 7
Registered: ‎06-14-2017

Re: Oozie Spark Action java.lang.NoClassDefFoundError: org/apache/spark/Logging

@ yshi
Highlighted
Cloudera Employee
Posts: 35
Registered: ‎08-18-2014

Re: Oozie Spark Action java.lang.NoClassDefFoundError: org/apache/spark/Logging

@kaidata


I prefer to communicate via the community, so that everyone can share in the knowledge. 
Please open up a new topic for your specific question. This way, the whole community can participate, and you can get the benefit of everyone's experience.

Announcements