Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Issue with oozie workflow submitting sh to launch spark2 job

Issue with oozie workflow submitting sh to launch spark2 job

New Contributor

Let me explain the issue, it's turning me crazy...

I have a spark2 program that I want to submit from an oozie workflow.

Because spark2 by default cannot be submitted directly from oozie, I have created an sh with the spark2-submit sentence to execute the spark2 job.

If I run this sh from the console it works perfect. However, when I run it from the oozie workflow, there is no way to make it work, but the worse is I cannot see a clear error on logs returned by the execution.

These are the files I am using:

 

thintest.sh:

spark2-submit --master yarn --class main hdfs:///home/cloudera/thintest/thintest-assembly-0.1.jar

 

job.properties

oozie.use.system.libpath=True
dryrun=False
nameNode=hdfs://quickstart.cloudera:8020
jobTracker=quickstart.cloudera:8032
security_enabled=False
hiveXml=/etc/hive/conf/hive-site.xml
appName=thintest
appPath=${nameNode}/home/cloudera/thintest
oozie.wf.application.path=${appPath}/workflow.xml
shellPath=${appPath}/thintest.sh

 

workflow.xml

<?xml version="1.0" encoding="UTF-8"?>
<workflow-app name="thintest" xmlns="uri:oozie:workflow:0.5">

<start to="shell"/>

  <action name="shell">
    <shell xmlns="uri:oozie:shell-action:0.1">

      <job-tracker>${jobTracker}</job-tracker>
      <name-node>${nameNode}</name-node>

      <exec>${shellPath}</exec>
      <env-var>HADOOP_USER_NAME=${wf:user()}</env-var>
      <file>${shellPath}#thintest.sh</file> 

    </shell>

    <ok to="end"/>
    <error to="kill"/>

  </action>

  <kill name="kill">
    <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
  </kill>

  <end name="end"/>
</workflow-app>

Any help will be very appreciated.