Support Questions
Find answers, ask questions, and share your expertise

Running Spark workflows from Hue/Oozie




We are using cdh 5.6 and want to run spark jobs using oozie workflows.

The problems are when we select the jar files. we have the option to browse for jar file form HDFS, but when we run the workflow, we receive : File file:/user/tlapusan/oozie/jars/test/ExtractMatcherPoints-1.0-SNAPSHOT.jar does not exist

I know that I can add the complete path for jar file, like ${namenode}/path/to/jar but it's inconvinient because :

1. we need to export our workflows to other clusters.

2. we have HA HDFS and the manually namenode can be changed.


It's also confusing because in MapReduce workflow it read the jar file fron HDFS by default .


Do you know how to set HDFS as a default filesystem for Spark workflows ?