Community Articles
Find and share helpful community-sourced technical articles
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (1)
Cloudera Employee

The article describes how to install and configure Sparkmagic to run in HDP2.5 against Livy Server and Spark 1.6.2


Using Jupyter with Sparkmagic and Livy Server on HDP 2.5 in HCC

  • 1) Install Jupyter

###### Init venv
## Just first time
sudo yum install python-pip python-dev python-virtualenv -y
mkdir ~/jupyter_env

## After 2nd
virtualenv --system-site-packages ~/jupyter_env
source ~/jupyter_env/bin/activate

curl -O
sudo python
sudo easy_install -U pip

python -m pip install --upgrade pip
# for jupyter, it need gcc
sudo yum install gcc -y
sudo pip install jupyter notebook ipython


  • 2) Use Jupyter to connect Spark Livy

 #  Failed building wheel for pykerberos 
sudo yum install krb5-devel -y

sudo -H pip install sparkmagic

sudo  pip install  hdijupyterutils 
sudo  pip install autovizwidget
sudo  pip install sparkmagic
pip show sparkmagic
pip show autovizwidget
cd /usr/lib/python2.7/site-packages 
jupyter-kernelspec install --user sparkmagic/kernels/sparkkernel 
jupyter-kernelspec install --user sparkmagic/kernels/pysparkkernel
sudo -H jupyter nbextension enable --py --sys-prefix widgetsnbextension
  • 3) Start Notebook
cd ~/
jupyter notebook --ip=
  • 4) Connect from Jupyter to remote Spark cluster


In[ ]:  %load_ext sparkmagic.magics
In[ ]:  %manage_spark


Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
2 of 2
Last update:
‎08-17-2019 08:23 AM
Updated by:
Top Kudoed Authors