Support Questions

Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

Oozie Spark2 workflow error

Expert Contributor

Hi. I try simple Spark2 Oozie Workflow example Even though I followed Hortonworks Spark2 Configuration Post I couldn't succeed Spark2 workflow. I intentionally made simple Python code to make sure the error not caused by action itself. But I am helpless I could't achieve simple Oozie Spark2 workflow. Anyone seccesfull Oozie Spark2 workflow?

My job properties:

nameNode=hdfs://NameNode_FQDN
jobTracker=hdfs_FQDN:8050
queueName=default
examplesRoot=examples
oozie.use.system.libpath=true
oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/spark2
master=yarn-cluster
oozie.action.sharelib.for.spark=spark2

My work flow:

<workflow-app xmlns='uri:oozie:workflow:0.5' name='SparkPythonPi'>
          <start to='spark-node' />
          
          <action name='spark-node'>
            <spark xmlns="uri:oozie:spark-action:0.1">
              <job-tracker>${jobTracker}</job-tracker>
              <name-node>${nameNode}</name-node>
              <master>${master}</master>
              <name>Python-Spark-Pi</name>
              <jar>pi.py</jar>
            </spark>
            <ok to="end" />
            <error to="fail" />
          </action>
          
          <kill name="fail">
            <message>Workflow failed, error message [${wf:errorMessage(wf:lastErrorNode())}]</message>
          </kill>
          <end name='end' />
        </workflow-app>

My python file under lib directory:

sc.version

Oozie Server Error screenshot is attached.

YARN logs

yarn logs -applicationId <application_ID> 
ontainer exited with a non-zero exit code 1
Failing this attempt. Failing the application.
  ApplicationMaster host: N/A
  ApplicationMaster RPC port: -1
  queue: default
  start time: 1507200668162
  final status: FAILED
  tracking URL: <URL>
  user: oozie

<<< Invocation of Main class completed <<<


Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, Application application_<ID> finished with failed status
org.apache.spark.SparkException: Application application_<ID> finished with failed status

40689-oozie-error.png

2 REPLIES 2

Can you please attach the logs application id "application_1506602683416_0253"

Please run below command to get the logs

yarn logs -applicationId application_1506602683416_0253

Expert Contributor

I run many after every workflow try but logs either didn't help.Thank you.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.