question Using Spark2 on HDP 2.5 in Archives of Support Questions (Read Only)

Using Spark2 on HDP 2.5

aravind_kanda — Wed, 08 Mar 2017 05:20:29 GMT

Hello, I followed the below tutorial to install Jupyter and all is fine:

https://hortonworks.com/hadoop-tutorial/using-ipython-notebook-with-apache-spark/

Now I want to use Spark 2 instead. I changed the spark_home to the /usr/hdp/current/spark2-client folder and trying to run the start_ipython_notebook.sh command. It says

IPYTHON and IPYTHON_OPTS are removed in Spark 2.0+. Remove these from the environment and set PYSPARK_DRIVER_PYTHON and PYSPARK_DRIVER_PYTHON_OPTS instead.

Can someone suggest what change should I make to the ipython startup script?

Re: Using Spark2 on HDP 2.5

aervits — Wed, 08 Mar 2017 16:35:58 GMT

From https://github.com/apache/spark/blob/master/bin/pyspark

# In Spark 2.0, IPYTHON and IPYTHON_OPTS are removed and pyspark fails to launch if either option is set in the user's environment. Instead, users should set PYSPARK_DRIVER_PYTHON=ipython to use IPython and set PYSPARK_DRIVER_PYTHON_OPTS to pass options when starting the Python driver# (e.g. PYSPARK_DRIVER_PYTHON_OPTS='notebook').  This supports full customization of the IPython# and executor Python executables.

Re: Using Spark2 on HDP 2.5

aravind_kanda — Thu, 09 Mar 2017 02:23:31 GMT

Just what i was looking for. thanks!

Re: Using Spark2 on HDP 2.5

fishjhu — Mon, 19 Aug 2019 07:54:19 GMT

@Arvind Kandaswamy, can you please share the code that you used to solve the problem? It is not working for me.

Re: Using Spark2 on HDP 2.5

amod400 — Wed, 25 Apr 2018 14:48:15 GMT

Please someone please answer