This worked!     I already made these changes prior to running the last command.   hdp-select status hadoop-client     Set a couple of parameters export HADOOP_OPTS="-Dhdp.version=2.6.1.0-129” export HADOOP_CONF_DIR=/etc/hadoop/conf   Source-in the environment source ~/get_env.sh       Included last two lines to $SPARK_HOME/conf/spark-defaults.conf spark.driver.extraJavaOptions   -Dhdp.version=2.6.1.0-129 spark.yarn.am.extraJavaOptions -Dhdp.version=2.6.1.0-129   Added Hadoop version under Ambari / Yarn / Advanced / Custom:  hdp.version=2.6.1.0-129     Ensure this runs okay  yarn jar hadoop-mapreduce-examples.jar pi 5 5   Run spark pi example under yarn   cd /home/spark/spark-2.4.4-bin-hadoop2.7   spark-submit   --class org.apache.spark.examples.SparkPi  --master yarn   --deploy-mode cluster   --executor-memory 2G  --num-executors 5  --executor-cores 2   --conf spark.authenticate.enableSaslEncryption=true   --conf spark.network.sasl.serverAlwaysEncrypt=true   --conf spark.authenticate=true   examples/jars/spark-examples_2.11-2.4.4.jar   100  ... View more

That looks really promising, thanks for the link! ... View more

Thanks for your response. I'll follow that course. Cheers ... View more

Ah, thanks. Feeling very silly I left that param in. 🙂 ... View more

Thanks, that makes sense and the upgrade went smoothly. ... View more

Hi: I need to know if i can delete from database the extra row, also the skip_sc_failures column with value =1 is error or ok?? ambari=> select * from upgrade; upgrade_id | cluster_id | request_id | from_version | to_version | direction | downgrade_allowed | skip_failures | skip_sc_failures | upgrade_pac kage | upgrade_type ------------+------------+------------+--------------+--------------+-----------+-------------------+---------------+------------------+--------------- ---------+-------------- 1 | 2 | 820 | 2.3.2.0-2950 | 2.4.0.0-169 | UPGRADE | 1 | 1 | 0 | nonrolling-upg rade-2.4 | NON_ROLLING 2 | 2 | 821 | 2.3.2.0-2950 | 2.3.2.0-2950 | DOWNGRADE | 1 | 0 | 0 | nonrolling-upg rade-2.4 | NON_ROLLING 3 | 2 | 894 | 2.3.2.0-2950 | 2.4.0.0-169 | UPGRADE | 1 | 1 | 1 | nonrolling-upg rade-2.4 | NON_ROLLING ... View more

@Brenden Cobb Awesome!!!! 🙂 I will send the feedback to the support team. ... View more