Community Articles
Find and share helpful community-sourced technical articles
Labels (1)
Cloudera Employee

The article describes how to install and configure Sparkmagic to run in HDP2.5 against Livy Server and Spark 1.6.2


Using Jupyter with Sparkmagic and Livy Server on HDP 2.5 in HCC

  • 1) Install Jupyter

###### Init venv
## Just first time
sudo yum install python-pip python-dev python-virtualenv -y
mkdir ~/jupyter_env

## After 2nd
virtualenv --system-site-packages ~/jupyter_env
source ~/jupyter_env/bin/activate

curl -O
sudo python
sudo easy_install -U pip

python -m pip install --upgrade pip
# for jupyter, it need gcc
sudo yum install gcc -y
sudo pip install jupyter notebook ipython


  • 2) Use Jupyter to connect Spark Livy

 #  Failed building wheel for pykerberos 
sudo yum install krb5-devel -y

sudo -H pip install sparkmagic

sudo  pip install  hdijupyterutils 
sudo  pip install autovizwidget
sudo  pip install sparkmagic
pip show sparkmagic
pip show autovizwidget
cd /usr/lib/python2.7/site-packages 
jupyter-kernelspec install --user sparkmagic/kernels/sparkkernel 
jupyter-kernelspec install --user sparkmagic/kernels/pysparkkernel
sudo -H jupyter nbextension enable --py --sys-prefix widgetsnbextension
  • 3) Start Notebook
cd ~/
jupyter notebook --ip=
  • 4) Connect from Jupyter to remote Spark cluster


In[ ]:  %load_ext sparkmagic.magics
In[ ]:  %manage_spark