Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.

simple hue/oozie shell script failing ... seems to fail when oozie is extracting script from HDFS

SOLVED Go to solution

simple hue/oozie shell script failing ... seems to fail when oozie is extracting script from HDFS

Explorer

hello,

I'm a oozie first time user trying to automate some of our spark jobs. They were failing so I figured I'd start with a simple script to establish we have things configured correctly.


Bash cmd line: $ echo -e "A|1|2|3\nB|4|5|6" | hadoop fs -put - /ARisk/tmp/test.txt
works as expected.

I created an HDFS file: /user/hue/oozie/workspaces/workflows/craigTest.sh
with only the above cmd line in there.

$ hadoop fs -chmod 777 /user/hue/oozie/workspaces/workflows/craigTest.sh


created an oozie workflow, shell script, click and select path to craigTest.sh above.

When I save and run the job I get a failure,

here is the log stdout:

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], main() threw exception, Cannot run program "craigTest.sh" (in directory "/data6/yarn/nm/usercache/hue/appcache/application_1476725022124_0158/container_1476725022124_0158_01_000002"): error=2, No such file or directory
java.io.IOException: Cannot run program "craigTest.sh" (in directory "/data6/yarn/nm/usercache/hue/appcache/application_1476725022124_0158/container_1476725022124_0158_01_000002"): error=2, No such file or directory
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
at org.apache.oozie.action.hadoop.ShellMain.execute(ShellMain.java:102)
at org.apache.oozie.action.hadoop.ShellMain.run(ShellMain.java:61)


so I try to navigate to see where the job would run ....

[root@wrlmr1 usercache]# pwd; ls -al
/data6/yarn/nm/usercache
total 8
drwxr-xr-x 2 yarn yarn 4096 Jul 25 16:03 .
drwxr-xr-x 5 yarn hadoop 4096 Jul 25 16:43 ..


nothing in the dir here .... do I have some kind of permission issue?

any pointers appreciated, this is CDH 5.8 out of the box.


Thanks,
Craig

1 ACCEPTED SOLUTION

Accepted Solutions

Re: simple hue/oozie shell script failing ... seems to fail when oozie is extracting script from HDF

Explorer

Hello,

this finally got resolved, I was missing the <file> tag in the workflow.xml

            <exec>/user/mrt1/oozie-oozi/craigTest.sh</exec>
            <file>/user/mrt1/oozie-oozi/craigTest.sh#craigTest.sh</file>

 

A colleague suggested adding the file tag and trying it, low and behold it worked. I then went back to Hue and saw the "files+" section added the full path there and it worked.

 

The error message kind of makes sense, but I would think Hue wouldn't let me create the shell action without the <file> tag. I guess it means never create a workflow.xml with only the <exec> tag.

 

Craig

1 REPLY 1

Re: simple hue/oozie shell script failing ... seems to fail when oozie is extracting script from HDF

Explorer

Hello,

this finally got resolved, I was missing the <file> tag in the workflow.xml

            <exec>/user/mrt1/oozie-oozi/craigTest.sh</exec>
            <file>/user/mrt1/oozie-oozi/craigTest.sh#craigTest.sh</file>

 

A colleague suggested adding the file tag and trying it, low and behold it worked. I then went back to Hue and saw the "files+" section added the full path there and it worked.

 

The error message kind of makes sense, but I would think Hue wouldn't let me create the shell action without the <file> tag. I guess it means never create a workflow.xml with only the <exec> tag.

 

Craig