Using HDP 3.1 and unable to run spark2 despite the clients being installed on all nodes (via Ambari), eg.
(venv) ➜ ~ spark
zsh: spark: command not found...
zsh: command not found: spark
(venv) ➜ ~ spark2
zsh: spark2: command not found...
zsh: command not found: spark2
Checking the filesystem, nothing seems to be related directly to any spark binaries:
(venv) ➜ ~ find / -name spark 2>&1 | grep -v "Permission denied"
/home/spark
/var/lib/smartsense/hst-agent/resources/collection-scripts/spark
/var/log/spark
/var/spool/mail/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/parse/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/optimizer/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/exec/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/common/jsonexplain/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hive/spark
/tmp/hadoop-unjar3014181574139383154/biz/k11i/xgboost/spark
/usr/hdp/3.1.0.0-78/spark2/examples/src/main/java/org/apache/spark
/usr/hdp/3.1.0.0-78/spark2/examples/src/main/scala/org/apache/spark
/usr/hdp/3.1.0.0-78/oozie/share/lib/spark
Anyone know where the spark binaries are for any given nodes?