Issue was master node couldn't reach https://archive.cloudera.com/, Issue was solved when IT allowed it ... View more

MapReduce jobs can be submitted with ease, as all they mostly require is the correct config on the classpath (such as under src/main/resources for Maven projects). Spark/PySpark greatly relies on its script tooling to submit to a remote cluster so it is a little more involved to achieve this. IntelliJ IDEA has a remote execution option in its run targets that can be configured to copy over the build jar and invoke any arbitrary command on an edge host. This can be combined with remote debugging perhaps to get equal experience as MR. Another option is to use a web interface based editor such as CDSW. ... View more

The share-lib in Oozie is modular, so it only adds necessary jars for each action type. The java action is the most generic of all action types, and therefore receives none of the other action type dependencies (such as hive, pig, distcp, spark, etc.). The article you've linked to carries an answer to the question of 'how do I further include jars from action type X into my action type Y', which I've quoted below for convenience: """ For example, if you want all Pig actions in one of your Workflows to include the HCatalog ShareLib, you would add oozie.action.sharelib.for.pig=pig,hcatalog to your job.properties. """ So in your case, you may want to try and add: oozie.action.sharelib.for.java=java,hive ... View more