Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
Labels (1)
avatar
Expert Contributor

The article describes how to install and configure Sparkmagic to run in HDP2.5 against Livy Server and Spark 1.6.2

Reference:

Using Jupyter with Sparkmagic and Livy Server on HDP 2.5 in HCC

  • 1) Install Jupyter

http://jupyter.org/install

###### Init venv
## Just first time
sudo yum install python-pip python-dev python-virtualenv -y
mkdir ~/jupyter_env

## After 2nd
virtualenv --system-site-packages ~/jupyter_env
source ~/jupyter_env/bin/activate

curl -O  https://bootstrap.pypa.io/get-pip.py
sudo python get-pip.py
sudo easy_install -U pip

python -m pip install --upgrade pip
# for jupyter, it need gcc
sudo yum install gcc -y
sudo pip install jupyter notebook ipython


http://zzeng-hdp-ambari:8888/tree?token=17dfdcb7525ff7470a637752450bbd586f607eddccc86a7f

  • 2) Use Jupyter to connect Spark Livy

https://community.hortonworks.com/articles/70501/using-jupyter-with-sparkmagic-and-livy-server-on-h....

 #  Failed building wheel for pykerberos 
sudo yum install krb5-devel -y

sudo -H pip install sparkmagic

sudo  pip install  hdijupyterutils 
sudo  pip install autovizwidget
sudo  pip install sparkmagic
   
pip show sparkmagic
pip show autovizwidget
cd /usr/lib/python2.7/site-packages 
jupyter-kernelspec install --user sparkmagic/kernels/sparkkernel 
jupyter-kernelspec install --user sparkmagic/kernels/pysparkkernel
sudo -H jupyter nbextension enable --py --sys-prefix widgetsnbextension
  • 3) Start Notebook
cd ~/
jupyter notebook --ip=0.0.0.0
  • 4) Connect from Jupyter to remote Spark cluster

Readme: https://github.com/jupyter-incubator/sparkmagic


In[ ]:  %load_ext sparkmagic.magics
In[ ]:  %manage_spark

62915-qqnhh.png

6,536 Views
Version history
Last update:
‎08-17-2019 08:23 AM
Updated by:
Contributors