Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Where are the spark2 binaries?

Solved Go to solution
Highlighted

Where are the spark2 binaries?

Contributor

Using HDP 3.1 and unable to run spark2 despite the clients being installed on all nodes (via Ambari), eg. 

 

(venv) ➜ ~ spark
zsh: spark: command not found...
zsh: command not found: spark
(venv) ➜ ~ spark2
zsh: spark2: command not found...
zsh: command not found: spark2

 

Checking the filesystem, nothing seems to be related directly to any spark binaries:

 

(venv) ➜  ~ find / -name spark  2>&1 | grep -v "Permission denied"
/home/spark
/var/lib/smartsense/hst-agent/resources/collection-scripts/spark
/var/log/spark
/var/spool/mail/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/parse/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/optimizer/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/exec/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/common/jsonexplain/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hive/spark
/tmp/hadoop-unjar3014181574139383154/biz/k11i/xgboost/spark
/usr/hdp/3.1.0.0-78/spark2/examples/src/main/java/org/apache/spark
/usr/hdp/3.1.0.0-78/spark2/examples/src/main/scala/org/apache/spark
/usr/hdp/3.1.0.0-78/oozie/share/lib/spark

 

Anyone know where the spark binaries are for any given nodes?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Where are the spark2 binaries?

Contributor

Found the answer here: https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/running-spark-applications/content/running_...

The binaries appear to be in /usr/hdp/current/spark2-client/bin

Though note that the right way to refer to SPARK_HOME seems to be /usr/hdp/current/spark2-client

1 REPLY 1

Re: Where are the spark2 binaries?

Contributor

Found the answer here: https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/running-spark-applications/content/running_...

The binaries appear to be in /usr/hdp/current/spark2-client/bin

Though note that the right way to refer to SPARK_HOME seems to be /usr/hdp/current/spark2-client

Don't have an account?
Coming from Hortonworks? Activate your account here