I installed sandbox using Docker for Windows image.
Sandbox information: Created on: 01_02_2018_10_47_41 Hadoop stack version: Hadoop 22.214.171.124.6.4.0-91 Ambari Version: 126.96.36.199-143 Ambari Hash: 2989989d67edacff7e9db702b4cf0c080556dddc Ambari build: Release : 143 Java version: 1.8.0_161 OS Version: CentOS release 6.9 (Final)
In Zeppelin, when I attempt to use spark2 interpreter with pyspark
x = 1
I get the following error:
/usr/hdp/current/spark2-client/python/pyspark/context.py:205: UserWarning: Support for Python 2.6 is deprecated as of Spark 2.0.0 warnings.warn("Support for Python 2.6 is deprecated as of Spark 2.0.0") Traceback (most recent call last): File "/tmp/zeppelin_pyspark-3334147833533366579.py", line 302, in <module> __zeppelin__._setup_matplotlib() File "/tmp/zeppelin_pyspark-3334147833533366579.py", line 141, in _setup_matplotlib import backend_zinline File "/usr/hdp/current/zeppelin-server/interpreter/lib/python/backend_zinline.py", line 30, in <module> import mpl_config File "/usr/hdp/current/zeppelin-server/interpreter/lib/python/mpl_config.py", line 99, in <module> _init_config() File "/usr/hdp/current/zeppelin-server/interpreter/lib/python/mpl_config.py", line 83, in _init_config fmt = matplotlib.rcParams['savefig.format'] KeyError: 'savefig.format
In hdp shell if I type python --version it is 2.6.6
I am confused why the sandbox image would use a deprecated version of python. If it needs python3 to work properly, then python3 should be installed by default when the Virtual Machine is first created.
I tried installing python3 in the Centos OS and restarting the hdp services but without success.
Please could you give me step by step instructions to go from a fresh install on VMWare Workstation with the error detailed above to a state where I am able to use Zeppelin with spark2 and pyspark. It would be much appreciated. Thank you.
You need to follow many steps to make this works.
- I installed anaconda3 and the critical step is do not let anaconda3 to be configured in the environment variables. HDP platform needs python 2 for some scripts and the python path needs to be resolved to a python 2 installation.
Since I want to have spark and spark2 interpreters I commented the SPARK_HOME line in the zeppelin-env.sh file:
Another configuration I changed in this file:
According to the documentation, the variable ZEPPELIN_JAVA_OPTS changed in spark2 to ZEPPELIN_INTP_JAVA_OPTS. Since both versions are active these two variables are defined:
exportZEPPELIN_JAVA_OPTS="-Dhdp.version=None -Dspark.executor.memory=512m -Dspark.executor.instances=2 -Dspark.yarn.queue=default"
export ZEPPELIN_INTP_JAVA_OPTS="-Dhdp.version=None -Dspark.executor.memory=512m -Dspark.executor.instances=2 -Dspark.yarn.queue=default"
- You need to configure the spark2 interpreter as follow:
I also created a Python interpreter:
Finally I created a symbolic link to be able to find conda
Create symlink to /bin/conda: ln -s /opt/anaconda3/bin/conda /bin/conda
Of course you have to adjust the paths above to your paths.
Hope that helps.
Kind regards, Paul