Reply
New Contributor
Posts: 1
Registered: ‎10-22-2018

Oozie shell action not executing - Cannot run program "sparkSubmit.sh"

[ Edited ]

Hello Every body, I need some help.

I'm working on Cloudera Express 5.15.0
When I run a shell action with a Shell script for running spark application, I am constantly confronted with an error. These error seem like same but These are diffrent.
At the first time I met this error, There were just two directories 'usercache', 'filecache' under '/DATA01/yarn/nm/usercache/user_01'. And then I remove all directories '/DATA01/yarn/nm/usercache/user_01' on every each Data nodes.


Now, I realize that oozie refer different directory from real created directory.

Ooze refer :
/DATA01/yarn/nm/usercache/user_01/appcache/application_1540195922373_0063/container_1540195922373_0063_01_000003

 

But Server has :
/DATA01/yarn/nm/usercache/user_01/appcache/application_1540195922373_0063/container_1540195922373_0063_01_000002

 

There is no directory "container_1540195922373_0063_01_000003" on the Data Node.

Can I get some help ?

 

Error Log ===================================================================================

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], main() threw exception, Cannot run program "sparkSubmit.sh" (in directory "/DATA01/yarn/nm/usercache/user_01/appcache/application_1540195922373_0063/container_1540195922373_0063_01_000003"): error=2, No such file or directory
java.io.IOException: Cannot run program "sparkSubmit.sh" (in directory "/DATA01/yarn/nm/usercache/user_01/appcache/application_1540195922373_0063/container_1540195922373_0063_01_000003"): error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
at org.apache.oozie.action.hadoop.ShellMain.execute(ShellMain.java:111)
at org.apache.oozie.action.hadoop.ShellMain.run(ShellMain.java:70)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:81)
at org.apache.oozie.action.hadoop.ShellMain.main(ShellMain.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:235)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.io.IOException: error=2, No such file or directory
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
at java.lang.ProcessImpl.start(ProcessImpl.java:134)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
... 17 more

Announcements