Oozie shell action not executing - Cannot run program "sparkSubmit.sh"

[#1]

Hello Every body, I need some help.

I'm working on Cloudera Express 5.15.0
When I run a shell action with a Shell script for running spark application, I am constantly confronted with an error. These error seem like same but These are diffrent.
At the first time I met this error, There were just two directories 'usercache', 'filecache' under '/DATA01/yarn/nm/usercache/user_01'. And then I remove all directories '/DATA01/yarn/nm/usercache/user_01' on every each Data nodes.

Now, I realize that oozie refer different directory from real created directory.

Ooze refer :
/DATA01/yarn/nm/usercache/user_01/appcache/application_1540195922373_0063/container_1540195922373_0063_01_000003

But Server has :
/DATA01/yarn/nm/usercache/user_01/appcache/application_1540195922373_0063/container_1540195922373_0063_01_000002

There is no directory "container_1540195922373_0063_01_000003" on the Data Node.

Can I get some help ?

Error Log ===================================================================================

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], main() threw exception, Cannot run program "sparkSubmit.sh" (in directory "/DATA01/yarn/nm/usercache/user_01/appcache/application_1540195922373_0063/container_1540195922373_0063_01_000003"): error=2, No such file or directory
java.io.IOException: Cannot run program "sparkSubmit.sh" (in directory "/DATA01/yarn/nm/usercache/user_01/appcache/application_1540195922373_0063/container_1540195922373_0063_01_000003"): error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at org.apache.oozie.action.hadoop.ShellMain.execute(ShellMain.java:111)
at org.apache.oozie.action.hadoop.ShellMain.run(ShellMain.java:70)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:81)
at org.apache.oozie.action.hadoop.ShellMain.main(ShellMain.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:235)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 17 more

[#2]

Hi,

I am afraid if you are passing the shell script name under file tag as well or not and because of that oozie is not able to localize the shell script which is causing the error of "No such file or directory"

Can you please share the worfkflow.xml and job.properties file so we can check if you are passing it in right way or not.

Regards

Nitish

[#3]

Just remove unwanted characters line end of line in shell script. It will work.