This worked!     I already made these changes prior to running the last command.   hdp-select status hadoop-client     Set a couple of parameters export HADOOP_OPTS="-Dhdp.version=2.6.1.0-129” export HADOOP_CONF_DIR=/etc/hadoop/conf   Source-in the environment source ~/get_env.sh       Included last two lines to $SPARK_HOME/conf/spark-defaults.conf spark.driver.extraJavaOptions   -Dhdp.version=2.6.1.0-129 spark.yarn.am.extraJavaOptions -Dhdp.version=2.6.1.0-129   Added Hadoop version under Ambari / Yarn / Advanced / Custom:  hdp.version=2.6.1.0-129     Ensure this runs okay  yarn jar hadoop-mapreduce-examples.jar pi 5 5   Run spark pi example under yarn   cd /home/spark/spark-2.4.4-bin-hadoop2.7   spark-submit   --class org.apache.spark.examples.SparkPi  --master yarn   --deploy-mode cluster   --executor-memory 2G  --num-executors 5  --executor-cores 2   --conf spark.authenticate.enableSaslEncryption=true   --conf spark.network.sasl.serverAlwaysEncrypt=true   --conf spark.authenticate=true   examples/jars/spark-examples_2.11-2.4.4.jar   100  ... View more