Using Jupyter with Sparkmagic and Livy Server on HDP 2.6

[#1]

The article describes how to install and configure Sparkmagic to run in HDP2.5 against Livy Server and Spark 1.6.2

Reference:

Using Jupyter with Sparkmagic and Livy Server on HDP 2.5 in HCC

• 1) Install Jupyter

http://jupyter.org/install

###### Init venv
## Just first time
sudo yum install python-pip python-dev python-virtualenv -y
mkdir ~/jupyter_env

## After 2nd
virtualenv --system-site-packages ~/jupyter_env
source ~/jupyter_env/bin/activate

curl -O  https://bootstrap.pypa.io/get-pip.py
sudo python get-pip.py
sudo easy_install -U pip

python -m pip install --upgrade pip
# for jupyter, it need gcc
sudo yum install gcc -y
sudo pip install jupyter notebook ipython

http://zzeng-hdp-ambari:8888/tree?token=17dfdcb7525ff7470a637752450bbd586f607eddccc86a7f

• 2) Use Jupyter to connect Spark Livy

https://community.hortonworks.com/articles/70501/using-jupyter-with-sparkmagic-and-livy-server-on-h....

 #  Failed building wheel for pykerberos 
sudo yum install krb5-devel -y

sudo -H pip install sparkmagic

sudo  pip install  hdijupyterutils 
sudo  pip install autovizwidget
sudo  pip install sparkmagic
   
pip show sparkmagic
pip show autovizwidget
cd /usr/lib/python2.7/site-packages 
jupyter-kernelspec install --user sparkmagic/kernels/sparkkernel 
jupyter-kernelspec install --user sparkmagic/kernels/pysparkkernel
sudo -H jupyter nbextension enable --py --sys-prefix widgetsnbextension

• 3) Start Notebook

cd ~/
jupyter notebook --ip=0.0.0.0

• 4) Connect from Jupyter to remote Spark cluster

Readme: https://github.com/jupyter-incubator/sparkmagic

In[ ]:  %load_ext sparkmagic.magics

In[ ]:  %manage_spark