Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Oozie-Spark2 compatibility

Oozie-Spark2 compatibility

Contributor

I'm using HDP2.6.0 where I have installed 4.2.0v of Oozie and 2.1.1. of Spark.

My question is, can I manage and adjust those 2 so I can submit job of Spark2 application to Oozie or I need to wait for 5.0 version of Oozie. Does anyone had any of problems like this and how did he fix it?

5 REPLIES 5
Highlighted

Re: Oozie-Spark2 compatibility

Explorer

@Ivan Majnaric I think, you can manage and submit Spark2 applications to Oozie.

Here are the steps, I would like to suggest you:

  1. Create a new directory (spark2) into oozie sharelib and copy the spark2 jar into it.
  2. Update the Oozie sharelib using Oozie admin command.
  3. Now in you workflow job.properties file use the new spark2 library using "oozie.action.sharelib.for.spark=spark2". Because you will be specifying the spark2 application under spark xml element into workflow.xml
  4. It is possible to remove some jars from "oozie" directory of the oozie sharelib because of the jar version conflicts.

Hope this will help you and get you going. :)

Highlighted

Re: Oozie-Spark2 compatibility

New Contributor

Here are the steps to run spark2 jobs using oozie:

Configuring Oozie Spark Action for Spark 2

Highlighted

Re: Oozie-Spark2 compatibility

Expert Contributor

I followed word by word aforementiond post. But the result same: Failed or killed :)

Highlighted

Re: Oozie-Spark2 compatibility

Explorer

I followed instructions mentioned in Configuring Oozie Spark Action for Spark 2

And Injected the setting oozie.action.sharelib.for.spark=spark2 in job properties.

With the above mentioned setting, the job failed.

Once I added the setting in workflow.xml, the job succeeded.

<property>
    <name>oozie.action.sharelib.for.spark</name>
    <value>spark2</value>
</property>

P.S: Driver jar for the job was created with scope as provided for Spark2 dependencies.

Highlighted

Re: Oozie-Spark2 compatibility

New Contributor

1. Follow instructions from this site:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_spark-component-guide/content/ch_oozie-s...

2. Some of the libs that you'll have put in the spark2 sharelib directory on hdfs will probably conflct with libs from oozie sharelib directory. You will need to remove the conflicting libs from the spark2 sharelib directory.

3. After removing the conflicting libs, run oozie admin -sharelibupdate.

Don't have an account?
Coming from Hortonworks? Activate your account here