Support Questions
Find answers, ask questions, and share your expertise

Right pom.xml for HDP 2.5.3 / Spark 1.6.2. application written in Scala

With some googling I've created a pom.xml for a Scala Spark application targeted at HDP 2.5.3 / Spark 1.6.2, but I keep getting this exception when running it on a HDP 2.5.3 cluster:

"java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: scala/runtime/LambdaDeserialize"

Full project here | pom.xml only

What am I overlooking in the build settings? It looks like a dependency issue or something. I'm submitting the jar via this command:

spark-submit --class org.ward.Benchmark  --master yarn --deploy-mode cluster benchmark-1.0-SNAPSHOT-jar-with-dependencies.jar
2 REPLIES 2

Re: Right pom.xml for HDP 2.5.3 / Spark 1.6.2. application written in Scala

As per below post, can you try changing the scala version to 2.11 :

http://stackoverflow.com/questions/40499534/i-get-a-java-lang-noclassdeffounderror-when-i-try-to-run...

Re: Right pom.xml for HDP 2.5.3 / Spark 1.6.2. application written in Scala

Super Mentor

@Ward Bekker

I am able to run your attached Full project at my end successfully on HDP 2.5.3 / Spark 1.6.2

So looks like there is no change needed from the project side. Can you try running spark-submit from any other host.