failed to submit spark-submit in oozie shell action in CDH Hue

[#1]

Dear all,

I submited spark-submit(/opt/cloudera/parcels/CDH/lib/spark/bin/spark-submit) shell in oozie shell action by Hue workflow, it was failed with following error, and the workflow was killed by itselft.

Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]

The spark-submit command works well when I run it in CLI.

Can you help to give any suggestion?Thanks!

[#2]

I also tried to upload spark package into HDFS, and specified "spark-submit" in HDF, but it still had the "

Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]

" issue.

[#3]

Does anyone meet the similar issue?

[#4]

First of all you should look at the logs of the workflow.

Also when oozie run the command it is from one of the data-nodes.

Did you deploy the "spark gateway" on the data-nodes ?

I hope it will help you.

[#5]

I am running into similar issue. Below are the files :

job.properties

xyz-abc-ma-01:8020
oozie.use.system.libpath=True
security_enabled=False
dryrun=False
jobTracker=yarnRM
user.name=test
nameNode=hdfs://ns1
oozie.wf.application.path=${nameNode}/user/${user.name}/
oozie.wf.rerun.failnodes=true
oozieProjectRoot=.
queueName=default
oozie.libpath=${nameNode}/user/oozie/share/lib
oozie.use.system.libpath=true
oozie.wf.rerun.failnodes=true
oozieProjectRoot=${nameNode}/user/${user.name}/
appPath=${nameNode}/user/${user.name}/
oozie.wf.application.path=${appPath}
inputDir=${oozieProjectRoot}/data/*/*/*/*/*
lineCountShScriptPath=${appPath}/oozie-oozi/ss1.sh
lineCountShellScript=ss1.sh

workflow.xml

<workflow-app name="WorkFlowForShellActionWithCaptureOutput" xmlns="uri:oozie:workflow:0.1">
<start to="shellAction1"/>
<action name="shellAction1">
<shell xmlns="uri:oozie:shell-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>default</value>
</property>
</configuration>
<exec>${lineCountShellScript}</exec>
<file>${lineCountShScriptPath}#${lineCountShellScript}</file>
<capture-output/>
</shell>
<ok to="end"/>
<error to="end"/>
</action>
<kill name="killAction">
<message>"Killed job due to error"</message>
</kill>
<end name="end"/>
</workflow-app>

ss1.sh

#!/bin/sh
spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster --driver-memory 512m --executor-memory 512m /user/test/spark-examples.jar 10

I see this error - Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]

Following is log snippet. I am using Cloudera 5.7.

2016-07-13 01:48:36,556 INFO org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[tenant-Test-1-qaclustertest2-ed-01.xyz.com] USER[test] GROUP[-] TOKEN[] APP[WorkFlowForShellActionWithCaptureOutput] JOB[0000050-160708180818829-oozie-oozi-W] ACTION[0000050-160708180818829-oozie-oozi-W@shellAction1] action completed, external ID [job_1468015395911_0088]
2016-07-13 01:48:36,564 WARN org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[tenant-Test-1-qaclustertest2-ed-01.xyz.com] USER[test] GROUP[-] TOKEN[] APP[WorkFlowForShellActionWithCaptureOutput] JOB[0000050-160708180818829-oozie-oozi-W] ACTION[0000050-160708180818829-oozie-oozi-W@shellAction1] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
2016-07-13 01:48:36,608 INFO org.apache.oozie.command.wf.ActionEndXCommand: SERVER[tenant-Test-1-qaclustertest2-ed-01.xyz.com] USER[test] GROUP[-] TOKEN[] APP[WorkFlowForShellActionWithCaptureOutput] JOB[0000050-160708180818829-oozie-oozi-W] ACTION[0000050-160708180818829-oozie-oozi-W@shellAction1] ERROR is considered as FAILED for SLA
2016-07-13 01:48:36,647 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[tenant-Test-1-qaclustertest2-ed-01.xyz.com] USER[test] GROUP[-] TOKEN[] APP[WorkFlowForShellActionWithCaptureOutput] JOB[0000050-160708180818829-oozie-oozi-W] ACTION[0000050-160708180818829-oozie-oozi-W@end] Start action [0000050-160708180818829-oozie-oozi-W@end] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-07-13 01:48:36,650 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[tenant-Test-1-qaclustertest2-ed-01.xyz.com] USER[test] GROUP[-] TOKEN[] APP[WorkFlowForShellActionWithCaptureOutput] JOB[0000050-160708180818829-oozie-oozi-W] ACTION[0000050-160708180818829-oozie-oozi-W@end] [***0000050-160708180818829-oozie-oozi-W@end***]Action status=DONE

[#6]

Hi, the error tells you that the shell has exited with 1 (so there was an error during the shell execution).

But it do not gives you the shell log. The root cause of the issue should be found in the shell logs.

Get your eyes on them.

If you don't find them, try to reproduce the shell action manualy in order to get the console output ?

[#7]

I am facing a similiar issue was just wondering how did you resolve it.

[#8]

I have the same issue. Did you solve it? 

[#9]

have you solved this problem??

[#10]

For running a spark job you really should use a spark action.

It does it all for you and you do not have to figure out how to set up anything.

Wilfred