Support Questions

Find answers, ask questions, and share your expertise

Oozie not taking spark extraclasspath jars when running spark submit action

avatar

I am running spark-submit action in oozie. When i give spark.driver.extraClasspath or spark.executor.extraClasspath in spark-submit command it runs fine. But with oozie when i give those option in <spark-opts> tag, its not running.

For example

--conf "spark.executor.extraClassPath=/usr/hdp/current/phoenix-client/phoenix-4.7.0.2.6.3.0-235-spark2.jar:/usr/hdp/current/phoenix-client/phoenix-client.jar"

Above run fine in spark-submit command but not in oozie. In oozie if i copy those jars inside workflow/lib, it works fine.

even --file /etc/hbase/conf/hbase-site.xml also not working. I am passing hbase-site.xml from workflow/lib and its not the right way

Then what is the point having spark.executor.extraClasspath option in oozie?

1 REPLY 1

avatar
Expert Contributor

Try put <extraClassPath> settings into the global Spark config in Ambari instead, in the spark-defaults section (you may have to add them as customized).

This works for us with Cloudera and Spark 1.6.