Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Oozie Spark Job is failing Continuously on CDH 5.5.2

Highlighted

Oozie Spark Job is failing Continuously on CDH 5.5.2

Explorer

Hi, 

 

I am trying to run spark worklow in oozie. I have created workflow using Hue with Spark Action and given the jar, class details properly. It has been failing continously and really making frustrated. I tried all the hell of options with no use and i could not resolve in for a month now.

 

master: tried spark standalone master, yarn, local[*] and local, nothing worked out.

 

Question 1: For each and every spark job, it submits a MAPREDUCE and SPARK job. The job gets accepted and then both MR and Spark jobs shows up in Job Browser and after 20 to 25 minutes, they will fail and no logs will appear to debug. What should i do, where should i search and it has been really frustrating experience. 

 

I have tried with all the masters and all the options suggested over the internet.

 

workflow.xml

-----------------

 

<?xml version="1.0" encoding="UTF-8"?>
<workflow-app xmlns="uri:oozie:workflow:0.5" name="My_Workflow">
<global>
<configuration>
<property>
<name>spark.yarn.jar</name>
<value>${nameNode}/<JAR Location in HDFS></value>
</property>
</configuration>
</global>
<start to="spark-8df4" />
<kill name="Kill">
<message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="spark-8df4">
<spark xmlns="uri:oozie:spark-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<master>local[*]</master>
<mode>client</mode>
<name>MySpark</name>
<class>//  Main class of program  //</class>
<jar>${nameNode}/<JAR location></jar>
</spark>
<ok to="End" />
<error to="Kill" />
</action>
<end name="End" />
</workflow-app>

Don't have an account?
Coming from Hortonworks? Activate your account here