Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]
Created on
08-27-2019
02:58 PM
- last edited on
08-27-2019
07:32 PM
by
ask_bill_brooks
Using HDP 3.1 and unable to run spark2 despite the clients being installed on all nodes (via Ambari), eg.
(venv) ➜ ~ spark
zsh: spark: command not found...
zsh: command not found: spark
(venv) ➜ ~ spark2
zsh: spark2: command not found...
zsh: command not found: spark2
Checking the filesystem, nothing seems to be related directly to any spark binaries:
(venv) ➜ ~ find / -name spark 2>&1 | grep -v "Permission denied"
/home/spark
/var/lib/smartsense/hst-agent/resources/collection-scripts/spark
/var/log/spark
/var/spool/mail/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/parse/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/optimizer/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/ql/exec/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hadoop/hive/common/jsonexplain/spark
/tmp/hadoop-unjar3014181574139383154/org/apache/hive/spark
/tmp/hadoop-unjar3014181574139383154/biz/k11i/xgboost/spark
/usr/hdp/3.1.0.0-78/spark2/examples/src/main/java/org/apache/spark
/usr/hdp/3.1.0.0-78/spark2/examples/src/main/scala/org/apache/spark
/usr/hdp/3.1.0.0-78/oozie/share/lib/spark
Anyone know where the spark binaries are for any given nodes?
Created on 08-27-2019 03:03 PM - edited 08-27-2019 03:15 PM
Found the answer here: https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/running-spark-applications/content/running_...
The binaries appear to be in /usr/hdp/current/spark2-client/bin
Though note that the right way to refer to SPARK_HOME seems to be /usr/hdp/current/spark2-client
Created on 08-27-2019 03:03 PM - edited 08-27-2019 03:15 PM
Found the answer here: https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/running-spark-applications/content/running_...
The binaries appear to be in /usr/hdp/current/spark2-client/bin
Though note that the right way to refer to SPARK_HOME seems to be /usr/hdp/current/spark2-client