Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (1)
Cloudera Employee

The article describes how to install and configure Sparkmagic to run in HDP2.5 against Livy Server and Spark 1.6.2

Reference:

Using Jupyter with Sparkmagic and Livy Server on HDP 2.5 in HCC

  • 1) Install Jupyter

http://jupyter.org/install

###### Init venv
## Just first time
sudo yum install python-pip python-dev python-virtualenv -y
mkdir ~/jupyter_env

## After 2nd
virtualenv --system-site-packages ~/jupyter_env
source ~/jupyter_env/bin/activate

curl -O  https://bootstrap.pypa.io/get-pip.py
sudo python get-pip.py
sudo easy_install -U pip

python -m pip install --upgrade pip
# for jupyter, it need gcc
sudo yum install gcc -y
sudo pip install jupyter notebook ipython


http://zzeng-hdp-ambari:8888/tree?token=17dfdcb7525ff7470a637752450bbd586f607eddccc86a7f

  • 2) Use Jupyter to connect Spark Livy

https://community.hortonworks.com/articles/70501/using-jupyter-with-sparkmagic-and-livy-server-on-h....

 #  Failed building wheel for pykerberos 
sudo yum install krb5-devel -y

sudo -H pip install sparkmagic

sudo  pip install  hdijupyterutils 
sudo  pip install autovizwidget
sudo  pip install sparkmagic
   
pip show sparkmagic
pip show autovizwidget
cd /usr/lib/python2.7/site-packages 
jupyter-kernelspec install --user sparkmagic/kernels/sparkkernel 
jupyter-kernelspec install --user sparkmagic/kernels/pysparkkernel
sudo -H jupyter nbextension enable --py --sys-prefix widgetsnbextension
  • 3) Start Notebook
cd ~/
jupyter notebook --ip=0.0.0.0
  • 4) Connect from Jupyter to remote Spark cluster

Readme: https://github.com/jupyter-incubator/sparkmagic


In[ ]:  %load_ext sparkmagic.magics
In[ ]:  %manage_spark

62915-qqnhh.png

3,956 Views
Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
2 of 2
Last update:
‎08-17-2019 08:23 AM
Updated by:
 
Contributors
Top Kudoed Authors