Created on 10-05-2017 11:14 AM - edited 08-17-2019 08:30 PM
Hi. I try simple Spark2 Oozie Workflow example Even though I followed Hortonworks Spark2 Configuration Post I couldn't succeed Spark2 workflow. I intentionally made simple Python code to make sure the error not caused by action itself. But I am helpless I could't achieve simple Oozie Spark2 workflow. Anyone seccesfull Oozie Spark2 workflow?
My job properties:
nameNode=hdfs://NameNode_FQDN jobTracker=hdfs_FQDN:8050 queueName=default examplesRoot=examples oozie.use.system.libpath=true oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/spark2 master=yarn-cluster oozie.action.sharelib.for.spark=spark2
My work flow:
<workflow-app xmlns='uri:oozie:workflow:0.5' name='SparkPythonPi'> <start to='spark-node' /> <action name='spark-node'> <spark xmlns="uri:oozie:spark-action:0.1"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <master>${master}</master> <name>Python-Spark-Pi</name> <jar>pi.py</jar> </spark> <ok to="end" /> <error to="fail" /> </action> <kill name="fail"> <message>Workflow failed, error message [${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <end name='end' /> </workflow-app>
My python file under lib directory:
sc.version
Oozie Server Error screenshot is attached.
YARN logs
yarn logs -applicationId <application_ID> ontainer exited with a non-zero exit code 1 Failing this attempt. Failing the application. ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: default start time: 1507200668162 final status: FAILED tracking URL: <URL> user: oozie <<< Invocation of Main class completed <<< Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, Application application_<ID> finished with failed status org.apache.spark.SparkException: Application application_<ID> finished with failed status
Created 10-05-2017 02:23 PM
Can you please attach the logs application id "application_1506602683416_0253"
Please run below command to get the logs
yarn logs -applicationId application_1506602683416_0253
Created 10-24-2017 02:55 PM
I run many after every workflow try but logs either didn't help.Thank you.