Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Oozie shell action not executing - Cannot run program "sparkSubmit.sh"

avatar
New Contributor

Hello Every body, I need some help.

I'm working on Cloudera Express 5.15.0
When I run a shell action with a Shell script for running spark application, I am constantly confronted with an error. These error seem like same but These are diffrent.
At the first time I met this error, There were just two directories 'usercache', 'filecache' under '/DATA01/yarn/nm/usercache/user_01'. And then I remove all directories '/DATA01/yarn/nm/usercache/user_01' on every each Data nodes.


Now, I realize that oozie refer different directory from real created directory.

Ooze refer :
/DATA01/yarn/nm/usercache/user_01/appcache/application_1540195922373_0063/container_1540195922373_0063_01_000003

 

But Server has :
/DATA01/yarn/nm/usercache/user_01/appcache/application_1540195922373_0063/container_1540195922373_0063_01_000002

 

There is no directory "container_1540195922373_0063_01_000003" on the Data Node.

Can I get some help ?

 

Error Log ===================================================================================

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], main() threw exception, Cannot run program "sparkSubmit.sh" (in directory "/DATA01/yarn/nm/usercache/user_01/appcache/application_1540195922373_0063/container_1540195922373_0063_01_000003"): error=2, No such file or directory
java.io.IOException: Cannot run program "sparkSubmit.sh" (in directory "/DATA01/yarn/nm/usercache/user_01/appcache/application_1540195922373_0063/container_1540195922373_0063_01_000003"): error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at org.apache.oozie.action.hadoop.ShellMain.execute(ShellMain.java:111)
at org.apache.oozie.action.hadoop.ShellMain.run(ShellMain.java:70)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:81)
at org.apache.oozie.action.hadoop.ShellMain.main(ShellMain.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:235)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 17 more

2 REPLIES 2

avatar
Expert Contributor

Hi,

 

I am afraid if you are passing the shell script name under file tag as well or not and because of that oozie is not able to localize the shell script which is causing the error of "No such file or directory"

 

Can you please share the worfkflow.xml and job.properties file so we can check if you are passing it in right way or not.

 

Regards

Nitish

avatar
New Contributor

Just remove unwanted characters line end of line in shell script. It will work.