Reply
New Contributor
Posts: 1
Registered: ‎07-31-2015

CHD 5.4.0 Oozie unable to run Spark-submit - ClassNotFound

I have created a spark-submit Oozie workflow job and set the ShareLib property by setting oozie.use.system.libpath=true; just for sanity also 

ensured that hive-common.jar exists under "/user/oozie/share/lib/lib_20150528162838/hive" which has the class "org.apache.hadoop.hive.conf.HiveConf" in it.


Job definition:

<workflow-app name="MyLoad" xmlns="uri:oozie:workflow:0.5">
<start to="spark-1f56"/>
<kill name="Kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="spark-1f56">
<spark xmlns="uri:oozie:spark-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>oozie.use.system.libpath</name>
<value>true</value>
</property>
</configuration>
<master>yarn-client</master>
<mode>client</mode>
<name>MyLoader</name>
<class>com.bbb.mmmm.rrrr.MPoRETL</class>
<jar>lib/mpor_2.10-1.0.jar</jar>
<arg>MPoR_20150529.dat</arg>
<arg>20150531</arg>
</spark>
<ok to="End"/>
<error to="Kill"/>
</action>
<end name="End"/>
</workflow-app>


I am getting below error when I run the Oozie workflow job

 

ACTION[0000090-150722102506025-oozie-oozi-W@spark-1f56] Launcher exception: org/apache/hadoop/hive/conf/HiveConf
java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf

Cloudera Employee
Posts: 306
Registered: ‎01-16-2014

Re: CHD 5.4.0 Oozie unable to run Spark-submit - ClassNotFound

You need to make sure that you provide the hive-site.xml and all the hive dependencies correctly in the spark action. Hive on Spark and Spark SQL (which provides the HiveCOntext) are not officially supported yet. However: it can work but you need to do a number of things manually. Check the Spark forum for more information on how to provide the hive-site.xml to all nodes. The solution for that has been given on that forum.

 

Wilfred

Announcements