Reply
New Contributor
Posts: 1
Registered: ‎04-21-2016

NoSuchMethodError when running Spark in Oozie workflow

[ Edited ]

When trying to run an oozie workflow with a spark task, I get the following error:

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, org.apache.ivy.core.module.descriptor.DefaultModuleDescriptor.setDefaultConf(Ljava/lang/String;)V
java.lang.NoSuchMethodError: org.apache.ivy.core.module.descriptor.DefaultModuleDescriptor.setDefaultConf(Ljava/lang/String;)V
    at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:997)
    at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:286)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:153)

This appears to be a version mismatch between Spark 1.5.2 (which was built against ivy 2.4.0) and Oozie 4.1.0 (which comes with ivy 2.0.0-rc2).

Any thoughts on how to get around this issue?

Thanks,

Steve

New Contributor
Posts: 1
Registered: ‎06-22-2016

Re: NoSuchMethodError when running Spark in Oozie workflow

Hi Steve,

 

Were you able to resolve this issue? If yes, please share your solution. I am facing the same problem.

 

Thanks,

Aditya

New Contributor
Posts: 1
Registered: ‎11-02-2017

Re: NoSuchMethodError when running Spark in Oozie workflow

Hi,

 

   Yeah - I have the same issue - was this resolved? 

 

    I am tracking this at https://stackoverflow.com/questions/47045446/oozies-spark-submit-uses-ivy-2-4-method-missing-in-cdh-....

 

-=> codeaperature

Posts: 1,566
Kudos: 287
Solutions: 240
Registered: ‎07-31-2013

Re: NoSuchMethodError when running Spark in Oozie workflow

What CDH version are you using? For the original report, Spark 1.5.2 was not a base version of Spark in any CDH release, but 1.5.0 and 1.6.0 were present. Are you using a custom Spark version during the application packaging, instead of a CDH Spark dependency?: https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh5_maven_repo.html and https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh5_maven_repo_513x.h...
Backline Customer Operations Engineer
Announcements