Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.

CHD 5.4.0 Oozie unable to run Spark-submit - ClassNotFound

CHD 5.4.0 Oozie unable to run Spark-submit - ClassNotFound

New Contributor

I have created a spark-submit Oozie workflow job and set the ShareLib property by setting oozie.use.system.libpath=true; just for sanity also 

ensured that hive-common.jar exists under "/user/oozie/share/lib/lib_20150528162838/hive" which has the class "org.apache.hadoop.hive.conf.HiveConf" in it.


Job definition:

<workflow-app name="MyLoad" xmlns="uri:oozie:workflow:0.5">
<start to="spark-1f56"/>
<kill name="Kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="spark-1f56">
<spark xmlns="uri:oozie:spark-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>oozie.use.system.libpath</name>
<value>true</value>
</property>
</configuration>
<master>yarn-client</master>
<mode>client</mode>
<name>MyLoader</name>
<class>com.bbb.mmmm.rrrr.MPoRETL</class>
<jar>lib/mpor_2.10-1.0.jar</jar>
<arg>MPoR_20150529.dat</arg>
<arg>20150531</arg>
</spark>
<ok to="End"/>
<error to="Kill"/>
</action>
<end name="End"/>
</workflow-app>


I am getting below error when I run the Oozie workflow job

 

ACTION[0000090-150722102506025-oozie-oozi-W@spark-1f56] Launcher exception: org/apache/hadoop/hive/conf/HiveConf
java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf

1 REPLY 1

Re: CHD 5.4.0 Oozie unable to run Spark-submit - ClassNotFound

Super Collaborator

You need to make sure that you provide the hive-site.xml and all the hive dependencies correctly in the spark action. Hive on Spark and Spark SQL (which provides the HiveCOntext) are not officially supported yet. However: it can work but you need to do a number of things manually. Check the Spark forum for more information on how to provide the hive-site.xml to all nodes. The solution for that has been given on that forum.

 

Wilfred