Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Using Spark2 on HDP 2.5

Solved Go to solution

Using Spark2 on HDP 2.5

Contributor

Hello, I followed the below tutorial to install Jupyter and all is fine:

https://hortonworks.com/hadoop-tutorial/using-ipython-notebook-with-apache-spark/

Now I want to use Spark 2 instead. I changed the spark_home to the /usr/hdp/current/spark2-client folder and trying to run the start_ipython_notebook.sh command. It says

IPYTHON and IPYTHON_OPTS are removed in Spark 2.0+. Remove these from the environment and set PYSPARK_DRIVER_PYTHON and PYSPARK_DRIVER_PYTHON_OPTS instead.

Can someone suggest what change should I make to the ipython startup script?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Using Spark2 on HDP 2.5

Mentor

From https://github.com/apache/spark/blob/master/bin/pyspark

# In Spark 2.0, IPYTHON and IPYTHON_OPTS are removed and pyspark fails to launch if either option is set in the user's environment. Instead, users should set PYSPARK_DRIVER_PYTHON=ipython to use IPython and set PYSPARK_DRIVER_PYTHON_OPTS to pass options when starting the Python driver# (e.g. PYSPARK_DRIVER_PYTHON_OPTS='notebook').  This supports full customization of the IPython# and executor Python executables.
4 REPLIES 4

Re: Using Spark2 on HDP 2.5

Mentor

From https://github.com/apache/spark/blob/master/bin/pyspark

# In Spark 2.0, IPYTHON and IPYTHON_OPTS are removed and pyspark fails to launch if either option is set in the user's environment. Instead, users should set PYSPARK_DRIVER_PYTHON=ipython to use IPython and set PYSPARK_DRIVER_PYTHON_OPTS to pass options when starting the Python driver# (e.g. PYSPARK_DRIVER_PYTHON_OPTS='notebook').  This supports full customization of the IPython# and executor Python executables.

Re: Using Spark2 on HDP 2.5

Contributor

Just what i was looking for. thanks!

Re: Using Spark2 on HDP 2.5

New Contributor

@Arvind Kandaswamy, can you please share the code that you used to solve the problem? It is not working for me.

45807-capture.png

Re: Using Spark2 on HDP 2.5

New Contributor

Please someone please answer