Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

zeppelin pyspark cannot run with different minor version

Solved Go to solution
Highlighted

zeppelin pyspark cannot run with different minor version

Contributor

HDP 2.5.3 Spark 1.6.x.2.5 Zeppelin Notebook 0.6.0.2.5

Just installed zeppelin, trying to run some of the examples. On the zeppelin notebook server I have python 2.7 installed (/usr/bin/python). On the other nodes in the cluster I have python 2.6 (/usr/bin/python) and 2.7 (/usr/local/bin/python) installed.

I am runnning the "Hello World" example in zeppelin. Most steps run, but some fail with this error

Exception: Python in worker has different version 2.6 than that in driver 2.7, PySpark cannot run with different minor versions

In the stack trace I can see the node that I assume is the worker. It is not a spark client but I created /etc/spark/conf/spark-env.sh with the following

export PYSPARK_PYTHON=/usr/local/bin/python

export PYSPARK_DRIVER_PYTHON=python

pointing to python 2.7, the same as the version that is running on the Zeppelin notebook.

I am still getting the error. I am leery about removing the python 2.6 on the other nodes but am not sure how to get around this.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: zeppelin pyspark cannot run with different minor version

1 REPLY 1

Re: zeppelin pyspark cannot run with different minor version

Don't have an account?
Coming from Hortonworks? Activate your account here