https://community.cloudera.com/t5/Support-Questions/Which-jars-are-needed-for-ApplicationMaster/m-p/219148#M181049 <P><A rel="user" href="https://community.cloudera.com/users/11394/edday.html" nodeid="11394">@ed day</A></P><P>Sometimes this happens "spark-assembly" jar is not being deployed correctly in HDFS (or gets corrupted).<BR /><BR />So please try to Deploy the "spark-assembly" jar correctly in HDFS and removed users ".sparkStaging" directory </P><PRE># su - hdfs # hdfs dfs -rm /hdp/apps/x.x.x.x.x/spark/spark-hdp-assembly.jar # hdfs dfs -put /usr/hdp/x.x.x.x/spark/lib/spark-assembly-1.y.y.x.x.x.x-hadoopx.x.x.x.x..jar /hdp/apps/x.x.x.x/spark/spark-hdp-assembly.jar </PRE><P>.</P><P>Please replace the x.x.x.x with the version of hdp / component based on your setup.<BR />.</P><P><BR />And also please delete the<STRONG> ".sparkStaging"</STRONG> HDFS directory of the user who is running the Job.</P><PRE># su - testuser # hdfs dfs -rm -r -f .sparkStaging</PRE><P>.</P> Wed, 06 Dec 2017 18:16:22 GMT jsensharma 2017-12-06T18:16:22Z