I followed the steps from Eric's Blog to customize Oozie with Spark 2.   Best of luck with Spark 2! ... View more

Sparamas,   It's is really cool you want to run both versions of Spark. I personally decided to only use Spark 2. The setup is tricky but I have written a few instructions for you Spark 2 via Hue Shell Workflow in my blog. If I get to it and don't forget to do so, I will post it in GitHub.   If you skip the part to setup spark_submit2 as the default with alternatives, which sounds that is your case, then you must define spark_submit2 inside the shell scripts via Hue workflow.   I hope this helps you,   J. Levi ... View more

This definitely was the issue and the fix for me. I saw both jar files already in the java/lib/security dir and failed to replace them with the downloaded UnlimitedJCEPolicyJDK8 jar files.   It took me a long time to get back to replacing the files. After doing so, I restarted CMS successfuly.   Thanks for the post! ... View more

I'm running Spark2 submit command line successfully as local and yarn cluster mode in CDH 5.12.   I experience the same problem with saveAsTable when I run it in Hue Oozie workflow, given I loaded all Spark2 libraries to share/lib and pointed my workflow to that new dir. Since I have hundreds of tables, and some of them change structure over time, I am unable to declare Hive tables by hand.   In command line, Spark autogenerates the Hive table, as parquet, if it does not exist. Append mode also works well, given I have not tried the insert feature.     It is very tricky to run Spark2 cluster mode jobs. I made sure I entered first the spark-submit parameters first before my job arguments. See how I run the job below:       $ spark-submit --version version 2.2.0.cloudera2 Using Scala version 2.11.8 ... $ pwd /home/hg/app/spark/demo $ ls -1 ojdbc8.jar oracle_schema.properties demo_2.11-0.1.0-SNAPSHOT.jar $ spark-submit --verbose --class com.hernandezgraham.demo --master yarn --deploy-mode cluster --jars ojdbc8.jar --driver-class-path ojdbc8.jar --files oracle_schema.properties demo_2.11-0.1.0-SNAPSHOT.jar "argument1" "argument2" $ hdfs dfs -ls /user/hive/warehouse/demo drwxr-xr-x   - hdfs  hive          0 2018-02-12 09:39 /user/hive/warehouse/demo $ hive hive> show databases; OK default hive> show tables; OK demo hive> select * from demo limit 1; OK 1 hguser Time taken: 0.557 seconds, Fetched: 1 row(s)       Even though Spark 2 executes my code successfully in Oozie workflows, it still does not write the file and the Hive table. Perhaps that is a bug fix in 5.12 for the command line. The documentation is not quite clear for Hue. ... View more