Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Running Spark workflows from Hue/Oozie

Highlighted

Running Spark workflows from Hue/Oozie

Explorer

Hi,

 

We are using cdh 5.6 and want to run spark jobs using oozie workflows.

The problems are when we select the jar files. we have the option to browse for jar file form HDFS, but when we run the workflow, we receive : 

java.io.FileNotFoundException: File file:/user/tlapusan/oozie/jars/test/ExtractMatcherPoints-1.0-SNAPSHOT.jar does not exist

I know that I can add the complete path for jar file, like ${namenode}/path/to/jar but it's inconvinient because :

1. we need to export our workflows to other clusters.

2. we have HA HDFS and the manually namenode can be changed.

 

It's also confusing because in MapReduce workflow it read the jar file fron HDFS by default .

 

Do you know how to set HDFS as a default filesystem for Spark workflows ?

 

Thanks,

Tudor

 

Don't have an account?
Coming from Hortonworks? Activate your account here