Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NoSuchMethodError when running Spark in Oozie workflow

NoSuchMethodError when running Spark in Oozie workflow

New Contributor

When trying to run an oozie workflow with a spark task, I get the following error:

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, org.apache.ivy.core.module.descriptor.DefaultModuleDescriptor.setDefaultConf(Ljava/lang/String;)V
java.lang.NoSuchMethodError: org.apache.ivy.core.module.descriptor.DefaultModuleDescriptor.setDefaultConf(Ljava/lang/String;)V
    at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:997)
    at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:286)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:153)

This appears to be a version mismatch between Spark 1.5.2 (which was built against ivy 2.4.0) and Oozie 4.1.0 (which comes with ivy 2.0.0-rc2).

Any thoughts on how to get around this issue?

Thanks,

Steve

3 REPLIES 3

Re: NoSuchMethodError when running Spark in Oozie workflow

New Contributor

Hi Steve,

 

Were you able to resolve this issue? If yes, please share your solution. I am facing the same problem.

 

Thanks,

Aditya

Re: NoSuchMethodError when running Spark in Oozie workflow

New Contributor

Hi,

 

   Yeah - I have the same issue - was this resolved? 

 

    I am tracking this at https://stackoverflow.com/questions/47045446/oozies-spark-submit-uses-ivy-2-4-method-missing-in-cdh-....

 

-=> codeaperature

Re: NoSuchMethodError when running Spark in Oozie workflow

Master Guru
What CDH version are you using? For the original report, Spark 1.5.2 was not a base version of Spark in any CDH release, but 1.5.0 and 1.6.0 were present. Are you using a custom Spark version during the application packaging, instead of a CDH Spark dependency?: https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh5_maven_repo.html and https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh5_maven_repo_513x.h...