https://community.cloudera.com/t5/Support-Questions/I-am-unable-to-run-a-spark-job-successfully-using-Yarn-Rest/m-p/230944#M192791 <P>I have downloaded the latest hdp-sandbox docker. Here I am trying to run a spark job using Yarn Rest API approach.</P><P>I am using the following payload:</P><P> { "application-id":"application_1528874935802_0047", "application-name":"test", "am-container-spec": { "local-resources": { "entry": [ { "key":"AppMaster.jar", "value": { "resource":"hdfs://&lt;host&gt;:8020/user/spark-examples_2.11-2.2.0.2.6.4.0-91.jar", "type":"FILE", "visibility":"APPLICATION", "size": "43004", "timestamp": "1528878009810" } } ] }, "commands": { "command":"{{JAVA_HOME}}/bin/java -Xmx10m org.apache.spark.deploy.yarn.ApplicationMaster --class 'org.apache.spark.examples.SparkPi' --jar AppMaster.jar 1&gt;&lt;LOG_DIR&gt;/AppMaster.stdout 2&gt;&lt;LOG_DIR&gt;/AppMaster.stderr" }, "environment": { "entry": [ { "key": "DISTRIBUTEDSHELLSCRIPTTIMESTAMP", "value": "1528342427276" }, { "key": "CLASSPATH", "value": "{{CLASSPATH}}&lt;CPS&gt;./*&lt;CPS&gt;AppMaster.jar&lt;CPS&gt;{{HADOOP_CONF_DIR}}&lt;CPS&gt;/usr/hdp/current/hadoop-hdfs-client/lib/*&lt;CPS&gt;/usr/hdp/current/hadoop-hdfs-client/*&lt;CPS&gt;/usr/hdp/current/spark2-client/jars/*&lt;CPS&gt;{{HADOOP_YARN_HOME}}/*&lt;CPS&gt;{{HADOOP_YARN_HOME}}/lib/*&lt;CPS&gt;./log4j.properties" }, { "key": "DISTRIBUTEDSHELLSCRIPTLEN", "value": "6" } ] } }, "unmanaged-AM":"false", "max-app-attempts":"2", "resource": { "memory":"1024", "vCores":"1" }, "application-type":"YARN", "keep-containers-across-application-attempts":"false" }.</P><P>The spark job gets submitted but it fails with the folllowing error:</P><P><STRONG>Exception in thread "main" java.lang.ClassCastException: org.apache.hadoop.conf.Configuration cannot be cast to org.apache.hadoop.yarn.conf.YarnConfiguration</STRONG> at org.apache.spark.deploy.yarn.ApplicationMaster.&lt;init&gt;(ApplicationMaster.scala:61) at org.apache.spark.deploy.yarn.ApplicationMaster$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:767) at org.apache.spark.deploy.SparkHadoopUtil$anon$2.run(SparkHadoopUtil.scala:67) at org.apache.spark.deploy.SparkHadoopUtil$anon$2.run(SparkHadoopUtil.scala:66) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869) at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:66) at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:766) at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala) Failing this attempt. Failing the application.</P> Tue, 19 Jun 2018 18:45:58 GMT abhinavkumar219 2018-06-19T18:45:58Z