Cloudera Community

Welcome to the Cloudera Community

cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Advanced Search
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Who agreed with this topic

NoSuchMethodError when running Spark in Oozie workflow

avatar
author-rank stevekaeser
New Contributor

Created on ‎04-21-2016 10:40 AM - last edited on ‎11-03-2017 05:20 AM by Community Manager Community Manager

When trying to run an oozie workflow with a spark task, I get the following error:

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, org.apache.ivy.core.module.descriptor.DefaultModuleDescriptor.setDefaultConf(Ljava/lang/String;)V
java.lang.NoSuchMethodError: org.apache.ivy.core.module.descriptor.DefaultModuleDescriptor.setDefaultConf(Ljava/lang/String;)V
    at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:997)
    at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:286)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:153)

This appears to be a version mismatch between Spark 1.5.2 (which was built against ivy 2.4.0) and Oozie 4.1.0 (which comes with ivy 2.0.0-rc2).

Any thoughts on how to get around this issue?

Thanks,

Steve

2,971
2
Who agreed with this topic