Good guide! But needs some minor amendments: step 9: I think this is quite wrong, and will (try to) create links in the main HDP Spark2 installation (2.3). I did "ln -s /etc/hive/conf/hive-site.xml /usr/hdp/3.1.0.0-78/spark2.4/conf/hive-site.xml" step 10 :"Create HDFS directory": /spark2-history already will exist, and the "hadoop fs -mkdir /apps/spark2.4/warehouse" step needs a -p flag step 10, "Copy hadoop JARs": should be "cp /usr/hdp/3.1.0.0-78/hadoop/client/jersey-* /usr/hdp/3.1.0.0-78/spark2.4/jars/" (the JARs in the client dir, not lib/) After doing all this, I still ended up getting the "bad substitution" error I mentioned in https://community.hortonworks.com/questions/247162/run-spark-24-jobs-on-hdp-31.html I'll keep trying... ... View more