Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Oozie not taking spark extraclasspath jars when running spark submit action

avatar

I am running spark-submit action in oozie. When i give spark.driver.extraClasspath or spark.executor.extraClasspath in spark-submit command it runs fine. But with oozie when i give those option in <spark-opts> tag, its not running.

For example

--conf "spark.executor.extraClassPath=/usr/hdp/current/phoenix-client/phoenix-4.7.0.2.6.3.0-235-spark2.jar:/usr/hdp/current/phoenix-client/phoenix-client.jar"

Above run fine in spark-submit command but not in oozie. In oozie if i copy those jars inside workflow/lib, it works fine.

even --file /etc/hbase/conf/hbase-site.xml also not working. I am passing hbase-site.xml from workflow/lib and its not the right way

Then what is the point having spark.executor.extraClasspath option in oozie?

1 REPLY 1

avatar
Expert Contributor

Try put <extraClassPath> settings into the global Spark config in Ambari instead, in the spark-defaults section (you may have to add them as customized).

This works for us with Cloudera and Spark 1.6.