Support Questions
Find answers, ask questions, and share your expertise

Where are the spark2 binaries?

Expert Contributor

Using HDP 3.1 and unable to run spark2 despite the clients being installed on all nodes (via Ambari), eg. 

 

(venv) ➜ ~ spark
zsh: spark: command not found...
zsh: command not found: spark
(venv) ➜ ~ spark2
zsh: spark2: command not found...
zsh: command not found: spark2

 

Checking the filesystem, nothing seems to be related directly to any spark binaries:

 

(venv) ➜  ~ find / -name spark  2>&1 | grep -v "Permission denied"
/home/spark
/var/lib/smartsense/hst-agent/resources/collection-scripts/spark
/var/log/spark
/var/spool/mail/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/parse/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/optimizer/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/exec/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/common/jsonexplain/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hive/spark
/tmp/hadoop-unjar3014181574139383154/biz/k11i/xgboost/spark
/usr/hdp/3.1.0.0-78/spark2/examples/src/main/java/org/apache/spark
/usr/hdp/3.1.0.0-78/spark2/examples/src/main/scala/org/apache/spark
/usr/hdp/3.1.0.0-78/oozie/share/lib/spark

 

Anyone know where the spark binaries are for any given nodes?

1 ACCEPTED SOLUTION

Expert Contributor

Found the answer here: https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/running-spark-applications/content/running_...

The binaries appear to be in /usr/hdp/current/spark2-client/bin

Though note that the right way to refer to SPARK_HOME seems to be /usr/hdp/current/spark2-client

View solution in original post

1 REPLY 1

Expert Contributor

Found the answer here: https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/running-spark-applications/content/running_...

The binaries appear to be in /usr/hdp/current/spark2-client/bin

Though note that the right way to refer to SPARK_HOME seems to be /usr/hdp/current/spark2-client

; ;