11-10-2016 08:39 AM
I'm a oozie first time user trying to automate some of our spark jobs. They were failing so I figured I'd start with a simple script to establish we have things configured correctly.
Bash cmd line: $ echo -e "A|1|2|3\nB|4|5|6" | hadoop fs -put - /ARisk/tmp/test.txt
works as expected.
I created an HDFS file: /user/hue/oozie/workspaces/workflows/craigTest.sh
with only the above cmd line in there.
$ hadoop fs -chmod 777 /user/hue/oozie/workspaces/workflows/craigTest.sh
created an oozie workflow, shell script, click and select path to craigTest.sh above.
When I save and run the job I get a failure,
here is the log stdout:
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], main() threw exception, Cannot run program "craigTest.sh" (in directory "/data6/yarn/nm/usercache/hue/appcache/application_1476725022124_0158/container_1476725022124_0158_01_000002"): error=2, No such file or directory
java.io.IOException: Cannot run program "craigTest.sh" (in directory "/data6/yarn/nm/usercache/hue/appcache/application_1476725022124_0158/container_1476725022124_0158_01_000002"): error=2, No such file or directory
so I try to navigate to see where the job would run ....
[root@wrlmr1 usercache]# pwd; ls -al
drwxr-xr-x 2 yarn yarn 4096 Jul 25 16:03 .
drwxr-xr-x 5 yarn hadoop 4096 Jul 25 16:43 ..
nothing in the dir here .... do I have some kind of permission issue?
any pointers appreciated, this is CDH 5.8 out of the box.
11-14-2016 01:20 PM
this finally got resolved, I was missing the <file> tag in the workflow.xml
A colleague suggested adding the file tag and trying it, low and behold it worked. I then went back to Hue and saw the "files+" section added the full path there and it worked.
The error message kind of makes sense, but I would think Hue wouldn't let me create the shell action without the <file> tag. I guess it means never create a workflow.xml with only the <exec> tag.