yarn-site.xml - having entry like below <property> <name>yarn.nodemanager.aux-services.spark2_shuffle.classpath</name> <value>/usr/hdp/${hdp.version}/spark2/aux/*</value> </property> Make sure you have the jar file ("spark-{version}-yarn-shuffle.jar" ) in that above mentioned location. If not we can create a create directory and copy that file from any other location. Now it is working fine. ... View more