I am trying to install, config and run Jupyter Notebook on Hortonworks Docker Sandbox HDP with Centos 7. I have followed these steps.
I get the following error:
SPARK_MAJOR_VERSION is set to 2, using Spark2
Error in pyspark startup:
IPYTHON and IPYTHON_OPTS are removed in Spark 2.0+. Remove these from the environment and set PYSPARK_DRIVER_PYTHON and PYSPARK_DRIVER_PYTHON_OPTS instead.
I am not sure what this means or what to do. Where in Centos do I go to set environment variables like these mentioned in the error?
Hi @Paul Byrum,
Reading the tutorial I noted that on Step 7 the "--notebook-dir='/usr/hdp/220.127.116.11-2950/spark/'" is setted to Spark 1 and not Spark 2.
I suggest that you change your spark version "SPARK_MAJOR_VERSION=1" and try to start your script again.