Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Error in pyspark startup:

Error in pyspark startup:

Explorer

Hello,

I am trying to install, config and run Jupyter Notebook on Hortonworks Docker Sandbox HDP with Centos 7. I have followed these steps.

I followed these steps: https://community.hortonworks.com/articles/39128/tutorial-installconfigure-ipython-and-createrun-py....

/start_ipython_notebook.sh

I get the following error:

SPARK_MAJOR_VERSION is set to 2, using Spark2

Error in pyspark startup: IPYTHON and IPYTHON_OPTS are removed in Spark 2.0+. Remove these from the environment and set PYSPARK_DRIVER_PYTHON and PYSPARK_DRIVER_PYTHON_OPTS instead.

I am not sure what this means or what to do. Where in Centos do I go to set environment variables like these mentioned in the error?

Thanks,

Paul

5 REPLIES 5
Highlighted

Re: Error in pyspark startup:

Explorer

Hi @Paul Byrum,

Reading the tutorial I noted that on Step 7 the "--notebook-dir='/usr/hdp/2.3.2.0-2950/spark/'" is setted to Spark 1 and not Spark 2.

I suggest that you change your spark version "SPARK_MAJOR_VERSION=1" and try to start your script again.

Regards

Highlighted

Re: Error in pyspark startup:

Explorer

Thanks for your reply. I am new to Spark on Centos. What file do I make this change?

Highlighted

Re: Error in pyspark startup:

Explorer

On the shell, before execute your script.

export SPARK_MAJOR_VERSION=1;
./start_ipython_notebook.sh
Highlighted

Re: Error in pyspark startup:

Explorer

Thanks for the help. That solved my problem.

Highlighted

Re: Error in pyspark startup:

Explorer

Great.

Remember to set the best answer.

By

Don't have an account?
Coming from Hortonworks? Activate your account here