Spark Job Failing "Could not find or load main class org.apache.spark.deploy.yarn.ApplicationMaster"

[#1]

Hi,

When I'm running Sample Spark Job in client mode it executing and when I run the same job in cluster mode it's failing. May I know the reason.

Client mode:

./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --num-executors 1 --driver-memory 512m --executor-memory 512m --executor-cores 1 lib/spark-examples*.jar 10

Cluster mode:

./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --num-executors 3 --driver-memory 4g --executor-memory 2g --executor-cores 1 lib/spark-examples*.jar 10

Error message:

yarn logs -applicationId <applicationnumber> output:

Container: container_1466521315275_0219_02_000001 on hostname.domain.com_45454
==========================================================================================
LogType:stderr
Log Upload Time:Fri Jun 24 14:11:39 -0500 2016
LogLength:88
Log Contents:
Error: Could not find or load main class org.apache.spark.deploy.yarn.ApplicationMaster
End of LogType:stderr
LogType:stdout
Log Upload Time:Fri Jun 24 14:11:39 -0500 2016
LogLength:0
Log Contents:
End of LogType:stdout

Spark-default.conf file:

spark.driver.extraJavaOptions -Dhdp.version=2.3.2.0-2950
spark.history.kerberos.enabled true
spark.history.kerberos.keytab /etc/security/keytabs/spark.headless.keytab
spark.history.kerberos.principal spark-hdp@DOMAIN>COM
spark.history.provider org.apache.spark.deploy.yarn.history.YarnHistoryProvider
spark.history.ui.port 18080
spark.yarn.am.extraJavaOptions -Dhdp.version=2.3.2.0-2950
spark.yarn.containerLauncherMaxThreads 25
spark.yarn.driver.memoryOverhead 384
spark.yarn.executor.memoryOverhead 384
spark.yarn.historyServer.address sparkhistory.domain.com:18080
spark.yarn.max.executor.failures 3
spark.yarn.preserve.staging.files false
spark.yarn.queue default
spark.yarn.scheduler.heartbeat.interval-ms 5000
spark.yarn.services org.apache.spark.deploy.yarn.history.YarnHistoryService
spark.yarn.submit.file.replication 3

Any help is highly appreciated and thanks in advance.

[#2 · Accepted solution]

@SBandaru

If you are using spark with hdp, then you have to do following things.

1. Add these entries in your $SPARK_HOME/conf/spark-defaults.conf

   spark.driver.extraJavaOptions -Dhdp.version=2.2.0.0-2041 (your installed HDP version)

   spark.yarn.am.extraJavaOptions -Dhdp.version=2.2.0.0-2041 (your installed HDP version)

2. create java-opts file in $SPARK_HOME/conf and add the installed HDP version in that file like

-Dhdp.version=2.2.0.0-2041 (your installed HDP version)

to know hdp version please run command hdp-select status hadoop-client in the cluster

[#3]

This worked for me as well (when replacing Spark-1 with a private build based on Apache Spark)

[#4]

@SBandaru: i am facing the same issue. running the spark submit on local is working. However running the same in cluster mode from other instance is failing throwing the same error.

Are you able to resolve the issue, if so ca you please post the steps you did to resolve the issue!

[#5]

@Anil Reddy hi anil reddy am running into the same issue were able to solve this? Do we have to add any specific jars to the spark-submit command.

Thanks,

Ankush Reddy.

[#6]

Doing the two changes as said by @Amit Nandi above worked for me!