Oozie-Spark2 compatibility

[#1]

I'm using HDP2.6.0 where I have installed 4.2.0v of Oozie and 2.1.1. of Spark.

My question is, can I manage and adjust those 2 so I can submit job of Spark2 application to Oozie or I need to wait for 5.0 version of Oozie. Does anyone had any of problems like this and how did he fix it?

[#2]

@Ivan Majnaric I think, you can manage and submit Spark2 applications to Oozie.

Here are the steps, I would like to suggest you:

1. Create a new directory (spark2) into oozie sharelib and copy the spark2 jar into it.
2. Update the Oozie sharelib using Oozie admin command.
3. Now in you workflow job.properties file use the new spark2 library using "oozie.action.sharelib.for.spark=spark2". Because you will be specifying the spark2 application under spark xml element into workflow.xml
4. It is possible to remove some jars from "oozie" directory of the oozie sharelib because of the jar version conflicts.

Hope this will help you and get you going. 🙂

[#3]

Here are the steps to run spark2 jobs using oozie:

Configuring Oozie Spark Action for Spark 2

[#4]

I followed word by word aforementiond post. But the result same: Failed or killed 🙂

[#5]

I followed instructions mentioned in Configuring Oozie Spark Action for Spark 2

And Injected the setting oozie.action.sharelib.for.spark=spark2 in job properties.

With the above mentioned setting, the job failed.

Once I added the setting in workflow.xml, the job succeeded.

<property>
    <name>oozie.action.sharelib.for.spark</name>
    <value>spark2</value>
</property>

P.S: Driver jar for the job was created with scope as provided for Spark2 dependencies.

[#6]

1. Follow instructions from this site:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_spark-component-guide/content/ch_oozie-s...

2. Some of the libs that you'll have put in the spark2 sharelib directory on hdfs will probably conflct with libs from oozie sharelib directory. You will need to remove the conflicting libs from the spark2 sharelib directory.

3. After removing the conflicting libs, run oozie admin -sharelibupdate.