Support Questions
Find answers, ask questions, and share your expertise

Using Spark2 on HDP 2.5

Contributor

Hello, I followed the below tutorial to install Jupyter and all is fine:

https://hortonworks.com/hadoop-tutorial/using-ipython-notebook-with-apache-spark/

Now I want to use Spark 2 instead. I changed the spark_home to the /usr/hdp/current/spark2-client folder and trying to run the start_ipython_notebook.sh command. It says

IPYTHON and IPYTHON_OPTS are removed in Spark 2.0+. Remove these from the environment and set PYSPARK_DRIVER_PYTHON and PYSPARK_DRIVER_PYTHON_OPTS instead.

Can someone suggest what change should I make to the ipython startup script?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Using Spark2 on HDP 2.5

Mentor

From https://github.com/apache/spark/blob/master/bin/pyspark

# In Spark 2.0, IPYTHON and IPYTHON_OPTS are removed and pyspark fails to launch if either option is set in the user's environment. Instead, users should set PYSPARK_DRIVER_PYTHON=ipython to use IPython and set PYSPARK_DRIVER_PYTHON_OPTS to pass options when starting the Python driver# (e.g. PYSPARK_DRIVER_PYTHON_OPTS='notebook').  This supports full customization of the IPython# and executor Python executables.

View solution in original post

4 REPLIES 4

Re: Using Spark2 on HDP 2.5

Mentor

From https://github.com/apache/spark/blob/master/bin/pyspark

# In Spark 2.0, IPYTHON and IPYTHON_OPTS are removed and pyspark fails to launch if either option is set in the user's environment. Instead, users should set PYSPARK_DRIVER_PYTHON=ipython to use IPython and set PYSPARK_DRIVER_PYTHON_OPTS to pass options when starting the Python driver# (e.g. PYSPARK_DRIVER_PYTHON_OPTS='notebook').  This supports full customization of the IPython# and executor Python executables.

View solution in original post

Re: Using Spark2 on HDP 2.5

Contributor

Just what i was looking for. thanks!

Re: Using Spark2 on HDP 2.5

New Contributor

@Arvind Kandaswamy, can you please share the code that you used to solve the problem? It is not working for me.

45807-capture.png

Re: Using Spark2 on HDP 2.5

New Contributor

Please someone please answer