28788
DISCUSSIONS
102106
MEMBERS
3161
ARTICLES
Created on 04-21-2016 10:40 AM - last edited on 11-03-2017 05:20 AM by cjervis
When trying to run an oozie workflow with a spark task, I get the following error:
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, org.apache.ivy.core.module.descriptor.DefaultModuleDescriptor.setDefaultConf(Ljava/lang/String;)V
java.lang.NoSuchMethodError: org.apache.ivy.core.module.descriptor.DefaultModuleDescriptor.setDefaultConf(Ljava/lang/String;)V
at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:997)
at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:286)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:153)
This appears to be a version mismatch between Spark 1.5.2 (which was built against ivy 2.4.0) and Oozie 4.1.0 (which comes with ivy 2.0.0-rc2).
Any thoughts on how to get around this issue?
Thanks,
Steve