Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Oozie shell action not executing - Cannot run program "sparkSubmit.sh"

Highlighted

Oozie shell action not executing - Cannot run program "sparkSubmit.sh"

New Contributor

Hello Every body, I need some help.

I'm working on Cloudera Express 5.15.0
When I run a shell action with a Shell script for running spark application, I am constantly confronted with an error. These error seem like same but These are diffrent.
At the first time I met this error, There were just two directories 'usercache', 'filecache' under '/DATA01/yarn/nm/usercache/user_01'. And then I remove all directories '/DATA01/yarn/nm/usercache/user_01' on every each Data nodes.


Now, I realize that oozie refer different directory from real created directory.

Ooze refer :
/DATA01/yarn/nm/usercache/user_01/appcache/application_1540195922373_0063/container_1540195922373_0063_01_000003

 

But Server has :
/DATA01/yarn/nm/usercache/user_01/appcache/application_1540195922373_0063/container_1540195922373_0063_01_000002

 

There is no directory "container_1540195922373_0063_01_000003" on the Data Node.

Can I get some help ?

 

Error Log ===================================================================================

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], main() threw exception, Cannot run program "sparkSubmit.sh" (in directory "/DATA01/yarn/nm/usercache/user_01/appcache/application_1540195922373_0063/container_1540195922373_0063_01_000003"): error=2, No such file or directory
java.io.IOException: Cannot run program "sparkSubmit.sh" (in directory "/DATA01/yarn/nm/usercache/user_01/appcache/application_1540195922373_0063/container_1540195922373_0063_01_000003"): error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at org.apache.oozie.action.hadoop.ShellMain.execute(ShellMain.java:111)
at org.apache.oozie.action.hadoop.ShellMain.run(ShellMain.java:70)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:81)
at org.apache.oozie.action.hadoop.ShellMain.main(ShellMain.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:235)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 17 more

1 REPLY 1

Re: Oozie shell action not executing - Cannot run program "sparkSubmit.sh"

Expert Contributor

Hi,

 

I am afraid if you are passing the shell script name under file tag as well or not and because of that oozie is not able to localize the shell script which is causing the error of "No such file or directory"

 

Can you please share the worfkflow.xml and job.properties file so we can check if you are passing it in right way or not.

 

Regards

Nitish