Thank you for this link: trying this out.   (Yet) another bizarre thing happened with 7.1.x installation - after several attempts I was able to install and get the Cloudera Manager up with web UI.   Most of the services were unavailable even after manually starting them (either through 'service' command or through web UI). Upon observing settings, system activities and logs, I noticed Cloudera was using 6.x binaries AND there were no 7.x files (except for the client libraries).    Hoping to see 7.x running with minimal difficulties. ... View more

This worked!     I already made these changes prior to running the last command.   hdp-select status hadoop-client     Set a couple of parameters export HADOOP_OPTS="-Dhdp.version=2.6.1.0-129” export HADOOP_CONF_DIR=/etc/hadoop/conf   Source-in the environment source ~/get_env.sh       Included last two lines to $SPARK_HOME/conf/spark-defaults.conf spark.driver.extraJavaOptions   -Dhdp.version=2.6.1.0-129 spark.yarn.am.extraJavaOptions -Dhdp.version=2.6.1.0-129   Added Hadoop version under Ambari / Yarn / Advanced / Custom:  hdp.version=2.6.1.0-129     Ensure this runs okay  yarn jar hadoop-mapreduce-examples.jar pi 5 5   Run spark pi example under yarn   cd /home/spark/spark-2.4.4-bin-hadoop2.7   spark-submit   --class org.apache.spark.examples.SparkPi  --master yarn   --deploy-mode cluster   --executor-memory 2G  --num-executors 5  --executor-cores 2   --conf spark.authenticate.enableSaslEncryption=true   --conf spark.network.sasl.serverAlwaysEncrypt=true   --conf spark.authenticate=true   examples/jars/spark-examples_2.11-2.4.4.jar   100  ... View more