Set your download options (shown in image below), and click on the link next to "Download Spark" (i.e. "spark-2.0.0-bin-hadoop2.7.tgz"):
This will download the gzipped tarball to your computer.
Next, startup the HDP 2.5 Sandbox image within your virtual machine (either using VirtualBox or VMFusion). Once the image is booted, startup a Terminal session on your laptop and copy the tarball to the VM. Here is an example using the 'scp' (secure copy) command, although you can use any file copy mechanism.
This will copy the file to the 'root' user's home directory on the VM. Next, login (via ssh) to the VM:
ssh -p 2222 firstname.lastname@example.org
Once logged in, unzip the tarball with this command:
tar -xvzf spark-2.0.0-bin-hadoop2.7.tgz
You can now navigate to the "seed" directory already created for Spark 2.0, and move the contents from the unzipped tar file into the current directory:
mv ~/spark-2.0.0-bin-hadoop2.7/* .
Next, change the ownership of the new files to match the local directory:
chown -R root:root *
Now, setup the SPARK_HOME environment variable for this session (or permanently by adding it to ~/.bash_profile)
Let's create the config files that we can edit them to configure Spark in the "conf" directory.
cp spark-env.sh.template spark-env.sh
cp spark-defaults.conf.template spark-defaults.conf
Edit the config files with a text editor (like vi or vim), and make sure the following environment variables and/or parameters are set below. Add the following lines to the file 'spark-env.sh' and then save the file:
Now that your config files are setup, change directory back to your $SPARK_HOME:
Before running a Spark application, you need to change 2 YARN settings to enable Yarn to allocate enough memory to run the jobs on the Sandbox. To change the Yarn settings, login to the Ambari console (http://127.0.0.1:8080/), and click on the "YARN" service along the left-hand side of the screen. Once the YARN Summary page is drawn, find the "Config" tab along top and click on it. Scroll down and you will see the "Settings" (not Advanced). Change the settings described below:
Note: Use the Edit/pencil icon to set each parameter to the exact values
1) Memory Node (Memory allocated for all YARN containers on a node) = 7800MB
After editing these parameters, click on the green "Save" button above the settings in Ambari. You will now need to Restart all affected services (Note: a yellow "Restart" icon should show up once the config settings are saved by Ambari; you can click on that button and select "Restart all affected services"). It may be faster to navigate to the Hosts page via the Tab, click on the single host, and look for the "Restart" button there.
Make sure that YARN is restarted successfully. Below is an image showing the new YARN settings:
Finally, you are ready to run the packaged SparkPi example using Spark 2.0.
In order to run SparkPi on YARN (yarn-client mode), run the command below, which switches user to "spark" and uses spark-submit to launch the precompiled SparkPi example program: