Community Articles

Find and share helpful community-sourced technical articles.
Celebrating as our community reaches 100,000 members! Thank you!


HDP 2.3.x cluster, whether it is a multi-node cluster or a single-node HDP Sandbox.


The Spark 1.6 Technical Preview is provided in RPM and DEB package formats. The following instructions assume RPM packaging:

  1. Download the Spark 1.6 RPM repository:
    wget -nv -O /etc/yum.repos.d/HDP-TP.repo
    For installing on Ubuntu use the following:
  2. Install the Spark Package: Download the Spark 1.6 RPM (and pySpark, if desired) and set it up on your HDP 2.3 cluster:
    yum install <strong>spark</strong>_2_3_4_1_10-master -y

    If you want to use pySpark, install it as follows and make sure that Python is installed on all nodes.

    yum install <strong>spark</strong>_2_3_4_1_10-python -y

    The RPM installer will also download core Hadoop dependencies. It will create “spark” as an OS user, and it will create the /user/spark directory in HDFS.

  3. Set JAVA_HOME and SPARK_HOME: Make sure that you set JAVA_HOME before you launch the Spark Shell or thrift server.
    export JAVA_HOME=<path to JDK 1.8>

    The Spark install creates the directory where Spark binaries are unpacked (/usr/hdp/ Set the SPARK_HOME variable to this directory:

    export SPARK_HOME=/usr/hdp/
  4. Create hive-site in the Spark conf directory: As user root, create the file SPARK_HOME/conf/hive-site.xml. Edit the file to contain only the following configuration setting:
    <strong><!--Make sure that <value> points to the Hive Metastore URI in your cluster -->
    </strong><value>thrift://</value><description>URI for client to contact metastore server</description></property></configuration>

Run the Spark Pi Example

To test compute-intensive tasks in Spark, the Pi example calculates pi by “throwing darts” at a circle — it generates points in the unit square ((0,0) to (1,1)) and counts how many points fall within the unit circle within the square. The result approximates pi/4, which is used to estimate Pi.

  1. Change to your Spark directory and switch to the spark OS user:
    cd $SPARK_HOME
    su spark
  2. Run the Spark Pi example in yarn-client mode:
    ./bin/spark-submit --class org.apache.spark.examples.SparkPi--master yarn-client --num-executors 3--driver-memory 512m--executor-memory 512m--executor-cores 1 lib/spark-examples*.jar 10

    Note: The Pi job should complete without any failure messages. It should produce output similar to the following. Note the value of pi near the end of the output.

    15/12/1613:21:05 INFO DAGScheduler:Job0 finished: reduce at SparkPi.scala:36, took 4.313782 s
    <strong>Piis roughly 3.139492</strong>15/12/1613:21:05 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null}
New Contributor

Hi, I couldn't see Spark pre-installed on HDP 2.4. If so, how to enable it?


where to run these commands?

Version history
Last update:
‎03-05-2016 12:32 AM
Updated by: