Support Questions
Find answers, ask questions, and share your expertise

Oozie Spark2 workflow error

Expert Contributor

Hi. I try simple Spark2 Oozie Workflow example Even though I followed Hortonworks Spark2 Configuration Post I couldn't succeed Spark2 workflow. I intentionally made simple Python code to make sure the error not caused by action itself. But I am helpless I could't achieve simple Oozie Spark2 workflow. Anyone seccesfull Oozie Spark2 workflow?

My job properties:

nameNode=hdfs://NameNode_FQDN
jobTracker=hdfs_FQDN:8050
queueName=default
examplesRoot=examples
oozie.use.system.libpath=true
oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/spark2
master=yarn-cluster
oozie.action.sharelib.for.spark=spark2

My work flow:

<workflow-app xmlns='uri:oozie:workflow:0.5' name='SparkPythonPi'>
          <start to='spark-node' />
          
          <action name='spark-node'>
            <spark xmlns="uri:oozie:spark-action:0.1">
              <job-tracker>${jobTracker}</job-tracker>
              <name-node>${nameNode}</name-node>
              <master>${master}</master>
              <name>Python-Spark-Pi</name>
              <jar>pi.py</jar>
            </spark>
            <ok to="end" />
            <error to="fail" />
          </action>
          
          <kill name="fail">
            <message>Workflow failed, error message [${wf:errorMessage(wf:lastErrorNode())}]</message>
          </kill>
          <end name='end' />
        </workflow-app>

My python file under lib directory:

sc.version

Oozie Server Error screenshot is attached.

YARN logs

yarn logs -applicationId <application_ID> 
ontainer exited with a non-zero exit code 1
Failing this attempt. Failing the application.
  ApplicationMaster host: N/A
  ApplicationMaster RPC port: -1
  queue: default
  start time: 1507200668162
  final status: FAILED
  tracking URL: <URL>
  user: oozie

<<< Invocation of Main class completed <<<


Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, Application application_<ID> finished with failed status
org.apache.spark.SparkException: Application application_<ID> finished with failed status

40689-oozie-error.png

2 REPLIES 2

Can you please attach the logs application id "application_1506602683416_0253"

Please run below command to get the logs

yarn logs -applicationId application_1506602683416_0253

Expert Contributor

I run many after every workflow try but logs either didn't help.Thank you.