Executing a real simple spark job, just print one line.   The jar file is generated by sbt. It works fine with spark-submit command line. When executing from oozie (BY hue's UI, only specified the jar file, no arguments, no 3-party jars):   Always return System.exit(101). No matter the spark.core dependency is included or not.   Does some knows the meaning of error 101? Can't find it in the source code in SparkMain or SparkSubmit.   Stdout:   ....                  --conf                     spark.yarn.security.tokens.hive.enabled=false                     --conf                     spark.yarn.security.tokens.hbase.enabled=false                     --verbose                     /user/mrt1/test/oozie-test.jar ================================================================= >>> Invoking Spark class now >>> Intercepting System.exit(101) <<< Invocation of Main class completed <<< Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], exit code [101] Oozie Launcher failed, finishing Hadoop job gracefully Oozie Launcher, uploading action data to HDFS sequence file: hdfs://.....:8020/user/mrt1/oozie-oozi/0000142-160913091926881-oozie-oozi-W/spark-57f6--spark/action-data.seq Oozie Launcher ends ... View more

hello, I'm a oozie first time user trying to automate some of our spark jobs. They were failing so I figured I'd start with a simple script to establish we have things configured correctly. Bash cmd line: $ echo -e "A|1|2|3\nB|4|5|6" | hadoop fs -put - /ARisk/tmp/test.txt works as expected. I created an HDFS file: /user/hue/oozie/workspaces/workflows/craigTest.sh with only the above cmd line in there. $ hadoop fs -chmod 777 /user/hue/oozie/workspaces/workflows/craigTest.sh created an oozie workflow, shell script, click and select path to craigTest.sh above. When I save and run the job I get a failure, here is the log stdout: Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], main() threw exception, Cannot run program "craigTest.sh" (in directory "/data6/yarn/nm/usercache/hue/appcache/application_1476725022124_0158/container_1476725022124_0158_01_000002"): error=2, No such file or directory java.io.IOException: Cannot run program "craigTest.sh" (in directory "/data6/yarn/nm/usercache/hue/appcache/application_1476725022124_0158/container_1476725022124_0158_01_000002"): error=2, No such file or directory at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047) at org.apache.oozie.action.hadoop.ShellMain.execute(ShellMain.java:102) at org.apache.oozie.action.hadoop.ShellMain.run(ShellMain.java:61) so I try to navigate to see where the job would run .... [root@wrlmr1 usercache]# pwd; ls -al /data6/yarn/nm/usercache total 8 drwxr-xr-x 2 yarn yarn 4096 Jul 25 16:03 . drwxr-xr-x 5 yarn hadoop 4096 Jul 25 16:43 .. nothing in the dir here .... do I have some kind of permission issue? any pointers appreciated, this is CDH 5.8 out of the box. Thanks, Craig ... View more