Where are the spark2 binaries?

rvillanueva — Wed, 28 Aug 2019 02:32:08 GMT

Using HDP 3.1 and unable to run spark2 despite the clients being installed on all nodes (via Ambari), eg.

(venv) ➜ ~ spark zsh: spark: command not found... zsh: command not found: spark (venv) ➜ ~ spark2 zsh: spark2: command not found... zsh: command not found: spark2

Checking the filesystem, nothing seems to be related directly to any spark binaries:

(venv) ➜ ~ find / -name spark 2>&1 | grep -v "Permission denied" /home/spark /var/lib/smartsense/hst-agent/resources/collection-scripts/spark /var/log/spark /var/spool/mail/spark /tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/parse/spark /tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/optimizer/spark /tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/exec/spark /tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/common/jsonexplain/spark /tmp/hadoop-unjar3014181574139383154/org/apache/hive/spark /tmp/hadoop-unjar3014181574139383154/biz/k11i/xgboost/spark /usr/hdp/3.1.0.0-78/spark2/examples/src/main/java/org/apache/spark /usr/hdp/3.1.0.0-78/spark2/examples/src/main/scala/org/apache/spark /usr/hdp/3.1.0.0-78/oozie/share/lib/spark

Anyone know where the spark binaries are for any given nodes?

Re: Where are the spark2 binaries?

rvillanueva — Tue, 27 Aug 2019 22:15:44 GMT

Found the answer here: https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/running-spark-applications/content/running_sample_spark_2_x_applications.html

The binaries appear to be in /usr/hdp/current/spark2-client/bin

Though note that the right way to refer to SPARK_HOME seems to be /usr/hdp/current/spark2-client

question Re: Where are the spark2 binaries? in Support Questions

Where are the spark2 binaries?

Re: Where are the spark2 binaries?