Reply
Contributor
Posts: 39
Registered: ‎09-20-2014

HUE Oozie ShellMain error

[ Edited ]

Hi!

 

I am using CDH 5.4.7 and I am struggling with Oozie in HUE using Shell snippet. 

 

My script contains only one line:

spark-submit --class cz.tmobile.surovecv.desms.DESMS --master yarn-client /home/ovasicek/DESMS_Parser-1.3-SNAPSHOT.jar $1 $2 $3 $4 $5

and every time I run the Oozie workflow, I am getting this error:

 Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]

and there is nothing helpful in the logs either.

 

I have put the .jar file into the workflow directory and .jar file is also accessible from the local disk. I don't know what can be wrong. If I run the exactly the same command from the command line, the command is successfully finished.

 

The stdout logs is ended by this:

>>> Invoking Shell command line now >>

Exit code of the Shell command 101
<<< Invocation of Shell command completed <<<


<<< Invocation of Main class completed <<<

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]

Oozie Launcher failed, finishing Hadoop job gracefully

Oozie Launcher, uploading action data to HDFS sequence file: hdfs://cloudcz.novalocal:8020/user/ovasicek/oozie-oozi/0000003-160217171518005-oozie-oozi-W/shell-f004--shell/action-data.seq

Oozie Launcher ends

 

Can you please help me?

 

Thank you very much!

Václav Surovec 

Cloudera Employee
Posts: 717
Registered: ‎07-30-2013

Re: HUE Oozie ShellMain error

Does the other Spark Oozie example work?

Did you look at the logs of the Oozie launcher (clicking on the log icon on
the action in the Oozie dashboard) or in Job Browser directly.

If you don't see any logs, how about in Oozie itself?
Contributor
Posts: 39
Registered: ‎09-20-2014

Re: HUE Oozie ShellMain error

Hi Romain,

 

the example Oozie job is working.

 

I was not able to see anything suspicious in the HUE logs except the part I have included in my first message.

 

I have checked the Oozie logs now and there is this below. Can you see something in it Romain? Thank you for your help in advance!

 

2016-02-18 23:22:26,701 WARN org.apache.oozie.util.ParameterVerifier: SERVER[cloudcz.novalocal] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] The application does not define formal parameters in its XML definition
2016-02-18 23:22:26,855 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[cloudcz.novalocal] USER[ovasicek] GROUP[-] TOKEN[] APP[DESMS_-_inputFilt] JOB[0000004-160218121234493-oozie-oozi-W] ACTION[0000004-160218121234493-oozie-oozi-W@:start:] Start action [0000004-160218121234493-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-02-18 23:22:26,856 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[cloudcz.novalocal] USER[ovasicek] GROUP[-] TOKEN[] APP[DESMS_-_inputFilt] JOB[0000004-160218121234493-oozie-oozi-W] ACTION[0000004-160218121234493-oozie-oozi-W@:start:] [***0000004-160218121234493-oozie-oozi-W@:start:***]Action status=DONE
2016-02-18 23:22:26,856 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[cloudcz.novalocal] USER[ovasicek] GROUP[-] TOKEN[] APP[DESMS_-_inputFilt] JOB[0000004-160218121234493-oozie-oozi-W] ACTION[0000004-160218121234493-oozie-oozi-W@:start:] [***0000004-160218121234493-oozie-oozi-W@:start:***]Action updated in DB!
2016-02-18 23:22:27,027 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[cloudcz.novalocal] USER[ovasicek] GROUP[-] TOKEN[] APP[DESMS_-_inputFilt] JOB[0000004-160218121234493-oozie-oozi-W] ACTION[0000004-160218121234493-oozie-oozi-W@shell-f004] Start action [0000004-160218121234493-oozie-oozi-W@shell-f004] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-02-18 23:22:27,739 WARN org.apache.hadoop.mapreduce.JobSubmitter: SERVER[cloudcz.novalocal] Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2016-02-18 23:22:27,772 WARN org.apache.hadoop.mapreduce.JobSubmitter: SERVER[cloudcz.novalocal] No job jar file set. User classes may not be found. See Job or Job#setJar(String).
2016-02-18 23:22:28,147 WARN org.apache.hadoop.mapreduce.v2.util.MRApps: SERVER[cloudcz.novalocal] cache file (mapreduce.job.cache.files) hdfs://cloudcz.novalocal:8020/user/hue/oozie/workspaces/hue-oozie-1454337978.29/lib/DESMS_Parser-1.3-SNAPSHOT.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://cloudcz.novalocal:8020/user/ovasicek/jars/DESMS_Parser-1.3-SNAPSHOT.jar This will be an error in Hadoop 2.0
2016-02-18 23:22:28,150 WARN org.apache.hadoop.mapreduce.v2.util.MRApps: SERVER[cloudcz.novalocal] cache file (mapreduce.job.cache.files) hdfs://cloudcz.novalocal:8020/user/hue/oozie/workspaces/hue-oozie-1454337978.29/lib/DESMS_Parser-1.3-SNAPSHOT.jar conflicts with cache file (mapreduce.job.cache.files) hdfs://cloudcz.novalocal:8020/user/oozie/share/lib/lib_20160202145019/oozie/DESMS_Parser-1.3-SNAPSHOT.jar This will be an error in Hadoop 2.0
2016-02-18 23:22:28,156 WARN org.apache.hadoop.mapreduce.v2.util.MRApps: SERVER[cloudcz.novalocal] cache file (mapreduce.job.cache.files) hdfs://cloudcz.novalocal:8020/user/ovasicek/scripts/spark-submit.sh conflicts with cache file (mapreduce.job.cache.files) hdfs://cloudcz.novalocal:8020/user/oozie/share/lib/lib_20160202145019/oozie/spark-submit.sh This will be an error in Hadoop 2.0
2016-02-18 23:22:28,287 INFO org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[cloudcz.novalocal] USER[ovasicek] GROUP[-] TOKEN[] APP[DESMS_-_inputFilt] JOB[0000004-160218121234493-oozie-oozi-W] ACTION[0000004-160218121234493-oozie-oozi-W@shell-f004] checking action, hadoop job ID [job_1455793885876_0005] status [RUNNING]
2016-02-18 23:22:28,289 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[cloudcz.novalocal] USER[ovasicek] GROUP[-] TOKEN[] APP[DESMS_-_inputFilt] JOB[0000004-160218121234493-oozie-oozi-W] ACTION[0000004-160218121234493-oozie-oozi-W@shell-f004] [***0000004-160218121234493-oozie-oozi-W@shell-f004***]Action status=RUNNING
2016-02-18 23:22:28,289 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[cloudcz.novalocal] USER[ovasicek] GROUP[-] TOKEN[] APP[DESMS_-_inputFilt] JOB[0000004-160218121234493-oozie-oozi-W] ACTION[0000004-160218121234493-oozie-oozi-W@shell-f004] [***0000004-160218121234493-oozie-oozi-W@shell-f004***]Action updated in DB!
2016-02-18 23:22:42,524 INFO org.apache.oozie.servlet.CallbackServlet: SERVER[cloudcz.novalocal] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000004-160218121234493-oozie-oozi-W] ACTION[0000004-160218121234493-oozie-oozi-W@shell-f004] callback for action [0000004-160218121234493-oozie-oozi-W@shell-f004]
2016-02-18 23:22:42,793 INFO org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[cloudcz.novalocal] USER[ovasicek] GROUP[-] TOKEN[] APP[DESMS_-_inputFilt] JOB[0000004-160218121234493-oozie-oozi-W] ACTION[0000004-160218121234493-oozie-oozi-W@shell-f004] action completed, external ID [job_1455793885876_0005]
2016-02-18 23:22:42,822 WARN org.apache.oozie.action.hadoop.ShellActionExecutor: SERVER[cloudcz.novalocal] USER[ovasicek] GROUP[-] TOKEN[] APP[DESMS_-_inputFilt] JOB[0000004-160218121234493-oozie-oozi-W] ACTION[0000004-160218121234493-oozie-oozi-W@shell-f004] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
2016-02-18 23:22:42,915 INFO org.apache.oozie.command.wf.ActionEndXCommand: SERVER[cloudcz.novalocal] USER[ovasicek] GROUP[-] TOKEN[] APP[DESMS_-_inputFilt] JOB[0000004-160218121234493-oozie-oozi-W] ACTION[0000004-160218121234493-oozie-oozi-W@shell-f004] ERROR is considered as FAILED for SLA
2016-02-18 23:22:43,021 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[cloudcz.novalocal] USER[ovasicek] GROUP[-] TOKEN[] APP[DESMS_-_inputFilt] JOB[0000004-160218121234493-oozie-oozi-W] ACTION[0000004-160218121234493-oozie-oozi-W@Kill] Start action [0000004-160218121234493-oozie-oozi-W@Kill] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-02-18 23:22:43,021 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[cloudcz.novalocal] USER[ovasicek] GROUP[-] TOKEN[] APP[DESMS_-_inputFilt] JOB[0000004-160218121234493-oozie-oozi-W] ACTION[0000004-160218121234493-oozie-oozi-W@Kill] [***0000004-160218121234493-oozie-oozi-W@Kill***]Action status=DONE
2016-02-18 23:22:43,021 INFO org.apache.oozie.command.wf.ActionStartXCommand: SERVER[cloudcz.novalocal] USER[ovasicek] GROUP[-] TOKEN[] APP[DESMS_-_inputFilt] JOB[0000004-160218121234493-oozie-oozi-W] ACTION[0000004-160218121234493-oozie-oozi-W@Kill] [***0000004-160218121234493-oozie-oozi-W@Kill***]Action updated in DB!

Contributor
Posts: 39
Registered: ‎09-20-2014

Re: HUE Oozie ShellMain error

So instead of .jar file which I created, I have used the Spark-Oozie example job in a bash file and run it using shell snippet in HUE, and I am getting still the same error..

 

command:

spark-submit --class org.apache.oozie.example.SparkFileCopy --master local file:///home/hdfs/oozie-examples.jar hdfs:///user/hdfs/testfile.txt hdfs:///user/hdfs/testfolder

 

Can you please help me? I have tried everything I don't know what can be wrong...

 

Thank you!

Explorer
Posts: 24
Registered: ‎12-05-2017

Re: HUE Oozie ShellMain error

how did you solved this problem? thanks

Contributor
Posts: 39
Registered: ‎09-20-2014

Re: HUE Oozie ShellMain error

Hello, I don't really remember. I think I have Spark spippet for Spark job instead of shell snippet and then it worked. Try that. Best, Václav

Highlighted
Explorer
Posts: 24
Registered: ‎12-05-2017

Re: HUE Oozie ShellMain error

ok! thanks!

Announcements