simple hue/oozie shell script failing ... seems to fail when oozie is extracting script from HDFS

[#1]

hello,

I'm a oozie first time user trying to automate some of our spark jobs. They were failing so I figured I'd start with a simple script to establish we have things configured correctly.

Bash cmd line: $ echo -e "A|1|2|3\nB|4|5|6" | hadoop fs -put - /ARisk/tmp/test.txt
works as expected.

I created an HDFS file: /user/hue/oozie/workspaces/workflows/craigTest.sh
with only the above cmd line in there.

$ hadoop fs -chmod 777 /user/hue/oozie/workspaces/workflows/craigTest.sh

created an oozie workflow, shell script, click and select path to craigTest.sh above.

When I save and run the job I get a failure,

here is the log stdout:

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], main() threw exception, Cannot run program "craigTest.sh" (in directory "/data6/yarn/nm/usercache/hue/appcache/application_1476725022124_0158/container_1476725022124_0158_01_000002"): error=2, No such file or directory
java.io.IOException: Cannot run program "craigTest.sh" (in directory "/data6/yarn/nm/usercache/hue/appcache/application_1476725022124_0158/container_1476725022124_0158_01_000002"): error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
at org.apache.oozie.action.hadoop.ShellMain.execute(ShellMain.java:102)
at org.apache.oozie.action.hadoop.ShellMain.run(ShellMain.java:61)

so I try to navigate to see where the job would run ....

[root@wrlmr1 usercache]# pwd; ls -al
/data6/yarn/nm/usercache
total 8
drwxr-xr-x 2 yarn yarn 4096 Jul 25 16:03 .
drwxr-xr-x 5 yarn hadoop 4096 Jul 25 16:43 ..

nothing in the dir here .... do I have some kind of permission issue?

any pointers appreciated, this is CDH 5.8 out of the box.

Thanks,
Craig

[#2 · Accepted solution]

Hello,

this finally got resolved, I was missing the <file> tag in the workflow.xml

            <exec>/user/mrt1/oozie-oozi/craigTest.sh</exec>
            <file>/user/mrt1/oozie-oozi/craigTest.sh#craigTest.sh</file>

A colleague suggested adding the file tag and trying it, low and behold it worked. I then went back to Hue and saw the "files+" section added the full path there and it worked.

The error message kind of makes sense, but I would think Hue wouldn't let me create the shell action without the <file> tag. I guess it means never create a workflow.xml with only the <exec> tag.

Craig