Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Configure Zeppelin Spark interpreter to work with python3

Highlighted

Configure Zeppelin Spark interpreter to work with python3

New Contributor

I'm using HDP-3.1 (spark2.3, python 3.6, Centos7)

 

I was able to configure it to work with python2.7, but with 3.6 I got the following error when running a notebook:

Exception: Python in worker has different version 2.7 than that in driver 3.6, PySpark cannot run with different minor versions.Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.

 

I have python 3.6 installed on all of the workers and environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set (see my configuration bellow):

 

Capture.PNG

 

Am I missing something?

2 REPLIES 2

Re: Configure Zeppelin Spark interpreter to work with python3

Mentor

@fklezin 

I think your zeppelin is not aware of the python version in the Zeppelin config at /usr/hdp/current/zeppelin-server/conf/interpreter.json, and change the below line 30 in the config:


"zeppelin.pyspark.python": {
"type": "string",
"name": "zeppelin.pyspark.python",
"value": "python"
},

To
"zeppelin.pyspark.python": {
"type": "string",
"name": "zeppelin.pyspark.python3",
"value": "python"
},


Make sure you have these values in Ambari UI--> Zeppelin-->Config-->Advanced zeppelin-env

export PYSPARK_PYTHON=python3
export PYSPARK_DRIVER_PYTHON=python3

 

Restart Zeppelin and retry

Re: Configure Zeppelin Spark interpreter to work with python3

Mentor

@fklezin 

 

Any updates do you still need help?

Don't have an account?
Coming from Hortonworks? Activate your account here