Support Questions

Find answers, ask questions, and share your expertise
Celebrating as our community reaches 100,000 members! Thank you!

Oozie not taking spark extraclasspath jars when running spark submit action


I am running spark-submit action in oozie. When i give spark.driver.extraClasspath or spark.executor.extraClasspath in spark-submit command it runs fine. But with oozie when i give those option in <spark-opts> tag, its not running.

For example

--conf "spark.executor.extraClassPath=/usr/hdp/current/phoenix-client/phoenix-"

Above run fine in spark-submit command but not in oozie. In oozie if i copy those jars inside workflow/lib, it works fine.

even --file /etc/hbase/conf/hbase-site.xml also not working. I am passing hbase-site.xml from workflow/lib and its not the right way

Then what is the point having spark.executor.extraClasspath option in oozie?


Expert Contributor

Try put <extraClassPath> settings into the global Spark config in Ambari instead, in the spark-defaults section (you may have to add them as customized).

This works for us with Cloudera and Spark 1.6.