Error in pyspark startup:

[#1]

Hello,

I am trying to install, config and run Jupyter Notebook on Hortonworks Docker Sandbox HDP with Centos 7. I have followed these steps.

I followed these steps: https://community.hortonworks.com/articles/39128/tutorial-installconfigure-ipython-and-createrun-py....

/start_ipython_notebook.sh

I get the following error:

SPARK_MAJOR_VERSION is set to 2, using Spark2

Error in pyspark startup: IPYTHON and IPYTHON_OPTS are removed in Spark 2.0+. Remove these from the environment and set PYSPARK_DRIVER_PYTHON and PYSPARK_DRIVER_PYTHON_OPTS instead.

I am not sure what this means or what to do. Where in Centos do I go to set environment variables like these mentioned in the error?

Thanks,

Paul

[#2]

Hi @Paul Byrum,

Reading the tutorial I noted that on Step 7 the "--notebook-dir='/usr/hdp/2.3.2.0-2950/spark/'" is setted to Spark 1 and not Spark 2.

I suggest that you change your spark version "SPARK_MAJOR_VERSION=1" and try to start your script again.

Regards

[#3]

Thanks for your reply. I am new to Spark on Centos. What file do I make this change?

[#4]

On the shell, before execute your script.

export SPARK_MAJOR_VERSION=1;
./start_ipython_notebook.sh

[#5]

Thanks for the help. That solved my problem.

[#6]

Great.

Remember to set the best answer.

By