With some googling I've created a pom.xml for a Scala Spark application targeted at HDP 2.5.3 / Spark 1.6.2, but I keep getting this exception when running it on a HDP 2.5.3 cluster:
"java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: scala/runtime/LambdaDeserialize"
Full project here | pom.xml only
What am I overlooking in the build settings? It looks like a dependency issue or something. I'm submitting the jar via this command:
spark-submit --class org.ward.Benchmark --master yarn --deploy-mode cluster benchmark-1.0-SNAPSHOT-jar-with-dependencies.jar
As per below post, can you try changing the scala version to 2.11 :
I am able to run your attached Full project at my end successfully on HDP 2.5.3 / Spark 1.6.2
So looks like there is no change needed from the project side. Can you try running spark-submit from any other host.