Support Questions
Find answers, ask questions, and share your expertise

Configure Zeppelin Spark interpreter to work with python3

New Contributor

I'm using HDP-3.1 (spark2.3, python 3.6, Centos7)

 

I was able to configure it to work with python2.7, but with 3.6 I got the following error when running a notebook:

Exception: Python in worker has different version 2.7 than that in driver 3.6, PySpark cannot run with different minor versions.Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.

 

I have python 3.6 installed on all of the workers and environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set (see my configuration bellow):

 

Capture.PNG

 

Am I missing something?

2 REPLIES 2

Mentor

@fklezin 

I think your zeppelin is not aware of the python version in the Zeppelin config at /usr/hdp/current/zeppelin-server/conf/interpreter.json, and change the below line 30 in the config:


"zeppelin.pyspark.python": {
"type": "string",
"name": "zeppelin.pyspark.python",
"value": "python"
},

To
"zeppelin.pyspark.python": {
"type": "string",
"name": "zeppelin.pyspark.python3",
"value": "python"
},


Make sure you have these values in Ambari UI--> Zeppelin-->Config-->Advanced zeppelin-env

export PYSPARK_PYTHON=python3
export PYSPARK_DRIVER_PYTHON=python3

 

Restart Zeppelin and retry

Mentor

@fklezin 

 

Any updates do you still need help?

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.