Created on 08-24-2015 08:00 PM - edited 09-16-2022 02:39 AM
HI
As you know, there is not supporting spark with 4.0.0-cdh5.3.2 oozie in cdh5.3.2.
But, we would like to get the function of workflow support.
How to resolve the issue in our cdh5.3.2 environment?
Thanks
Paul
Created 09-28-2015 08:08 PM
To rule out that we have a custom jar issue can you run the pi example to make sure that the cluster is (not) setup correctly?
We have documented how to run a spark application, with the example in our docs.
The error that you show points to a classpath error and you can not find the Spark classes on your class path.
WIlfred
Created 08-25-2015 02:02 AM
The only way to use Spark when you do not have a Spark action is to use the shell based action and create the proper spark-submit command for it.
You will need to make sure that the configuration and classpath etc is set from the action.
Wilfred
Created 08-25-2015 08:16 PM
Created 09-21-2015 02:49 AM
Created 09-21-2015 03:30 AM
Whatever you use as a spark-submit from the command line is what you use in the oozie shell action.
Make sure that you have the proper gateway for Spark and YARN installed on the oozie server so it has the configuration needed.
The rest works as if you have a standard oozie shell action (i.e. create the workflow, properties and shell script files) and place the files on the machine/hdfs so they can be found.
Wilfred
Created 09-22-2015 12:44 AM
Created 09-28-2015 08:08 PM
To rule out that we have a custom jar issue can you run the pi example to make sure that the cluster is (not) setup correctly?
We have documented how to run a spark application, with the example in our docs.
The error that you show points to a classpath error and you can not find the Spark classes on your class path.
WIlfred