Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

CHD 5.4.0 Oozie unable to run Spark-submit - ClassNotFound

CHD 5.4.0 Oozie unable to run Spark-submit - ClassNotFound

New Contributor

I have created a spark-submit Oozie workflow job and set the ShareLib property by setting oozie.use.system.libpath=true; just for sanity also 

ensured that hive-common.jar exists under "/user/oozie/share/lib/lib_20150528162838/hive" which has the class "org.apache.hadoop.hive.conf.HiveConf" in it.

Job definition:

<workflow-app name="MyLoad" xmlns="uri:oozie:workflow:0.5">
<start to="spark-1f56"/>
<kill name="Kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
<action name="spark-1f56">
<spark xmlns="uri:oozie:spark-action:0.1">
<ok to="End"/>
<error to="Kill"/>
<end name="End"/>

I am getting below error when I run the Oozie workflow job


ACTION[0000090-150722102506025-oozie-oozi-W@spark-1f56] Launcher exception: org/apache/hadoop/hive/conf/HiveConf
java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf


Re: CHD 5.4.0 Oozie unable to run Spark-submit - ClassNotFound

Super Collaborator

You need to make sure that you provide the hive-site.xml and all the hive dependencies correctly in the spark action. Hive on Spark and Spark SQL (which provides the HiveCOntext) are not officially supported yet. However: it can work but you need to do a number of things manually. Check the Spark forum for more information on how to provide the hive-site.xml to all nodes. The solution for that has been given on that forum.