Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Running Spark workflows from Hue/Oozie


Running Spark workflows from Hue/Oozie




We are using cdh 5.6 and want to run spark jobs using oozie workflows.

The problems are when we select the jar files. we have the option to browse for jar file form HDFS, but when we run the workflow, we receive : File file:/user/tlapusan/oozie/jars/test/ExtractMatcherPoints-1.0-SNAPSHOT.jar does not exist

I know that I can add the complete path for jar file, like ${namenode}/path/to/jar but it's inconvinient because :

1. we need to export our workflows to other clusters.

2. we have HA HDFS and the manually namenode can be changed.


It's also confusing because in MapReduce workflow it read the jar file fron HDFS by default .


Do you know how to set HDFS as a default filesystem for Spark workflows ?





Don't have an account?
Coming from Hortonworks? Activate your account here