Confused about the Spark setup doc

tli — Thu, 14 Jul 2016 04:11:50 GMT

Hi,

Looks like the highlighted parts below are conflicting. Both are trying to set SPARK_HOME variable (with different values). Do you have any ideas?

3. Set JAVA_HOME and SPARK_HOME:

Make sure that you set JAVA_HOME before you launch the Spark Shell or thrift server.

export JAVA_HOME=<path to JDK 1.8>

The Spark install creates the directory where Spark binaries are unpacked to /usr/hdp/2.3.4.1-10/spark.

Set the SPARK_HOME variable to this directory:

export SPARK_HOME=/usr/hdp/2.3.4.1-10/spark/

4. Create hive-site in the Spark conf directory:

As user root, create the file SPARK_HOME/conf/hive-site.xml. Edit the file to contain only the following configuration setting:

<name>hive.metastore.uris</name>

<value>thrift://sandbox.hortonworks.com:9083</value>

<description>URI for client to contact metastore server</description>

</property></configuration>

Set SPARK_HOME

If you haven’t already, make sure to set SPARK_HOME before proceeding:

export SPARK_HOME=/usr/hdp/current/spark-client

Re: Confused about the Spark setup doc

jgeorge — Mon, 19 Aug 2019 09:55:00 GMT

Hi,

I don't think it matters as "/usr/hdp/current/spark-client" directory is a symlink to the "/usr/hdp/2.3.4.1-10/spark/" directory.

you can verify like below:

[In my screenshot hdp version is slightly different from yours]

Thanks

tli — Fri, 15 Jul 2016 00:19:26 GMT

Got it. Thanks for the answer!