Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Oozie not taking spark extraclasspath jars when running spark submit action

Oozie not taking spark extraclasspath jars when running spark submit action

I am running spark-submit action in oozie. When i give spark.driver.extraClasspath or spark.executor.extraClasspath in spark-submit command it runs fine. But with oozie when i give those option in <spark-opts> tag, its not running.

For example

--conf "spark.executor.extraClassPath=/usr/hdp/current/phoenix-client/phoenix-4.7.0.2.6.3.0-235-spark2.jar:/usr/hdp/current/phoenix-client/phoenix-client.jar"

Above run fine in spark-submit command but not in oozie. In oozie if i copy those jars inside workflow/lib, it works fine.

even --file /etc/hbase/conf/hbase-site.xml also not working. I am passing hbase-site.xml from workflow/lib and its not the right way

Then what is the point having spark.executor.extraClasspath option in oozie?

1 REPLY 1
Highlighted

Re: Oozie not taking spark extraclasspath jars when running spark submit action

Contributor

Try put <extraClassPath> settings into the global Spark config in Ambari instead, in the spark-defaults section (you may have to add them as customized).

This works for us with Cloudera and Spark 1.6.

Don't have an account?
Coming from Hortonworks? Activate your account here