Tutorial: Using IPython Notebook with Apache Spark - Issue Resolution

[#1]

Tutorial Link

SandboxVersion: HDP 2.3.2
AmbariVersion2.1.2
Hadoop stack version:Hadoop2.7.1.2.3.2.0-2950

-------------------------------------------

Issue 1: No available when executing command yum groupinstall “Development tools“

Resolution: This occurs when you copy and pasted the command. The double quotes used are wrong. Use " instead. There run the following command with the right double quotes:

yum groupinstall "Development tools" 

The same occurs with pip install “ipython[notebook]“

Instead run

pip install "ipython[notebook]"

---------------------------------------

Issue 2: No ~/.ipython/profile_pyspark found after executing command ipython profile create pyspark

Resolution:

IPython was updated to 4.0.0 which uses jupyter

1. Run jupyter notebook --generate-config

2. Then nano /root/.jupyter/jupyter_notebook_config.py

-------------------------------------------

Issue 3: - profile Error when executing "~/start_ipython_notebook.sh"

Resolution:

Use

IPYTHON_OPTS="notebook" pyspark

instead of

IPYTHON_OPTS="notebook –profile pyspark" pyspark

[#2]

These issues have been resolved and information updated in the latest IPython with Spark Tutorial

[#3]

This help me a lot thank you ! I just needed to add --allow-root.