I am running spark-submit action in oozie. When i give spark.driver.extraClasspath or spark.executor.extraClasspath in spark-submit command it runs fine. But with oozie when i give those option in <spark-opts> tag, its not running.
Above run fine in spark-submit command but not in oozie. In oozie if i copy those jars inside workflow/lib, it works fine.
even --file /etc/hbase/conf/hbase-site.xml also not working. I am passing hbase-site.xml from workflow/lib and its not the right way
Then what is the point having spark.executor.extraClasspath option in oozie?
Try put <extraClassPath> settings into the global Spark config in Ambari instead, in the spark-defaults section (you may have to add them as customized).
This works for us with Cloudera and Spark 1.6.