Reply
Contributor
Posts: 41
Registered: ‎02-23-2016

failed to submit spark-submit in oozie shell action in CDH Hue

Dear all,

 

I submited spark-submit(/opt/cloudera/parcels/CDH/lib/spark/bin/spark-submit) shell in oozie shell action by Hue workflow, it was failed with following error, and the workflow was killed by itselft.

 

Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]

 

 

The spark-submit command works well when I run it in CLI.

 

Can you help to give any suggestion?Thanks!

 

 

Contributor
Posts: 41
Registered: ‎02-23-2016

Re: failed to submit spark-submit in oozie shell action in CDH Hue

[ Edited ]

I also tried to upload spark package into HDFS, and specified "spark-submit" in HDF, but it still had the "

Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]

" issue.

Highlighted
Contributor
Posts: 41
Registered: ‎02-23-2016

Re: failed to submit spark-submit in oozie shell action in CDH Hue

Does anyone meet the similar issue?

Posts: 176
Topics: 8
Kudos: 21
Solutions: 19
Registered: ‎07-16-2015

Re: failed to submit spark-submit in oozie shell action in CDH Hue

First of all you should look at the logs of the workflow.

 

Also when oozie run the command it is from one of the data-nodes.

Did you deploy the "spark gateway" on the data-nodes ?

 

I hope it will help you.

New Contributor
Posts: 1
Registered: ‎07-12-2016

Re: failed to submit spark-submit in oozie shell action in CDH Hue

[ Edited ]

I am running into similar issue. Below are the files :

 

job.properties

 

xyz-abc-ma-01:8020
oozie.use.system.libpath=True
security_enabled=False
dryrun=False
jobTracker=yarnRM
user.name=test
nameNode=hdfs://ns1
oozie.wf.application.path=${nameNode}/user/${user.name}/
oozie.wf.rerun.failnodes=true
oozieProjectRoot=.
queueName=default
oozie.libpath=${nameNode}/user/oozie/share/lib
oozie.use.system.libpath=true
oozie.wf.rerun.failnodes=true
oozieProjectRoot=${nameNode}/user/${user.name}/
appPath=${nameNode}/user/${user.name}/
oozie.wf.application.path=${appPath}
inputDir=${oozieProjectRoot}/data/*/*/*/*/*
lineCountShScriptPath=${appPath}/oozie-oozi/ss1.sh
lineCountShellScript=ss1.sh

 

workflow.xml

 

<workflow-app name="WorkFlowForShellActionWithCaptureOutput" xmlns="uri:oozie:workflow:0.1">
<start to="shellAction1"/>
<action name="shellAction1">
<shell xmlns="uri:oozie:shell-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>default</value>
</property>
</configuration>
<exec>${lineCountShellScript}</exec>
<file>${lineCountShScriptPath}#${lineCountShellScript}</file>
<capture-output/>
</shell>
<ok to="end"/>
<error to="end"/>
</action>
<kill name="killAction">
<message>"Killed job due to error"</message>
</kill>
<end name="end"/>
</workflow-app>

 

 

ss1.sh

 

#!/bin/sh
spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster --driver-memory 512m --executor-memory 512m /user/test/spark-examples.jar 10

 

I see this error - Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]

 

Following is log snippet. I am using Cloudera 5.7.

 

 


2016-07-13 01:48:36,556 INFO org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[tenant-Test-1-qaclustertest2-ed-01.xyz.com] USER[test] GROUP[-] TOKEN[] APP[WorkFlowForShellActionWithCaptureOutput] JOB[0000050-160708180818829-oozie-oozi-W] ACTION[0000050-160708180818829-oozie-oozi-W@shellAction1] action completed, external ID [job_1468015395911_0088]
2016-07-13 01:48:36,564 WARN org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[tenant-Test-1-qaclustertest2-ed-01.xyz.com] USER[test] GROUP[-] TOKEN[] APP[WorkFlowForShellActionWithCaptureOutput] JOB[0000050-160708180818829-oozie-oozi-W] ACTION[0000050-160708180818829-oozie-oozi-W@shellAction1] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
2016-07-13 01:48:36,608 INFO org.apache.oozie.command.wf.ActionEndXCommand: SERVER[tenant-Test-1-qaclustertest2-ed-01.xyz.com] USER[test] GROUP[-] TOKEN[] APP[WorkFlowForShellActionWithCaptureOutput] JOB[0000050-160708180818829-oozie-oozi-W] ACTION[0000050-160708180818829-oozie-oozi-W@shellAction1] ERROR is considered as FAILED for SLA
2016-07-13 01:48:36,647 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[tenant-Test-1-qaclustertest2-ed-01.xyz.com] USER[test] GROUP[-] TOKEN[] APP[WorkFlowForShellActionWithCaptureOutput] JOB[0000050-160708180818829-oozie-oozi-W] ACTION[0000050-160708180818829-oozie-oozi-W@end] Start action [0000050-160708180818829-oozie-oozi-W@end] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-07-13 01:48:36,650 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[tenant-Test-1-qaclustertest2-ed-01.xyz.com] USER[test] GROUP[-] TOKEN[] APP[WorkFlowForShellActionWithCaptureOutput] JOB[0000050-160708180818829-oozie-oozi-W] ACTION[0000050-160708180818829-oozie-oozi-W@end] [***0000050-160708180818829-oozie-oozi-W@end***]Action status=DONE

 

 

 

Posts: 176
Topics: 8
Kudos: 21
Solutions: 19
Registered: ‎07-16-2015

Re: failed to submit spark-submit in oozie shell action in CDH Hue

Hi, the error tells you that the shell has exited with 1 (so there was an error during the shell execution).

But it do not gives you the shell log. The root cause of the issue should be found in the shell logs.

 

Get your eyes on them.

 

If you don't find them, try to reproduce the shell action manualy in order to get the console output ?

Explorer
Posts: 34
Registered: ‎02-01-2017

Re: failed to submit spark-submit in oozie shell action in CDH Hue

I am facing a similiar issue was just wondering how did you resolve it.

New Contributor
Posts: 1
Registered: ‎04-03-2017

Re: failed to submit spark-submit in oozie shell action in CDH Hue

I have the same issue. Did you solve it? 

Explorer
Posts: 7
Registered: ‎12-05-2017

Re: failed to submit spark-submit in oozie shell action in CDH Hue

have you solved this problem??
Announcements