Support Questions

Find answers, ask questions, and share your expertise
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

Oozie not taking spark extraclasspath jars when running spark submit action

I am running spark-submit action in oozie. When i give spark.driver.extraClasspath or spark.executor.extraClasspath in spark-submit command it runs fine. But with oozie when i give those option in <spark-opts> tag, its not running.

For example

--conf "spark.executor.extraClassPath=/usr/hdp/current/phoenix-client/phoenix-"

Above run fine in spark-submit command but not in oozie. In oozie if i copy those jars inside workflow/lib, it works fine.

even --file /etc/hbase/conf/hbase-site.xml also not working. I am passing hbase-site.xml from workflow/lib and its not the right way

Then what is the point having spark.executor.extraClasspath option in oozie?



Try put <extraClassPath> settings into the global Spark config in Ambari instead, in the spark-defaults section (you may have to add them as customized).

This works for us with Cloudera and Spark 1.6.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.