Skip to main content
rvillanueva
Gold ⭐️
August 27, 2019
Solved

Where are the spark2 binaries?

Using HDP 3.1 and unable to run spark2 despite the clients being installed on all nodes (via Ambari), eg. 

 

(venv) ➜ ~ spark
zsh: spark: command not found...
zsh: command not found: spark
(venv) ➜ ~ spark2
zsh: spark2: command not found...
zsh: command not found: spark2

 

Checking the filesystem, nothing seems to be related directly to any spark binaries:

 

(venv) ➜  ~ find / -name spark  2>&1 | grep -v "Permission denied"
/home/spark
/var/lib/smartsense/hst-agent/resources/collection-scripts/spark
/var/log/spark
/var/spool/mail/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/parse/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/optimizer/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/exec/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/common/jsonexplain/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hive/spark
/tmp/hadoop-unjar3014181574139383154/biz/k11i/xgboost/spark
/usr/hdp/3.1.0.0-78/spark2/examples/src/main/java/org/apache/spark
/usr/hdp/3.1.0.0-78/spark2/examples/src/main/scala/org/apache/spark
/usr/hdp/3.1.0.0-78/oozie/share/lib/spark

 

Anyone know where the spark binaries are for any given nodes?

rvillanueva
Gold ⭐️
August 27, 2019

Found the answer here: https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/running-spark-applications/content/running_sample_spark_2_x_applications.html

The binaries appear to be in /usr/hdp/current/spark2-client/bin

Though note that the right way to refer to SPARK_HOME seems to be /usr/hdp/current/spark2-client

1 reply

rvillanueva
rvillanuevaAuthorAnswer
Gold ⭐️
August 27, 2019

Found the answer here: https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/running-spark-applications/content/running_sample_spark_2_x_applications.html

The binaries appear to be in /usr/hdp/current/spark2-client/bin

Though note that the right way to refer to SPARK_HOME seems to be /usr/hdp/current/spark2-client

Translate