question multiple versions of python issues in Archives of Support Questions (Read Only)

multiple versions of python issues

wazzu62 — Fri, 16 Sep 2022 13:41:49 GMT

I have two versions on python installed (2.6 and 2.7) Spark jobs run thru shell in pyspark are picking up one version of Python (2.7). Jobs submitted to the cluster via yarn are picking up the 2.6 version of python. How can I get yarn jobs to point to the 2.7 version?

Re: multiple versions of python issues

falbani — Wed, 12 Sep 2018 05:17:52 GMT

@Jon Page Try these before running spark-submit command:

export PYSPARK_DRIVER_PYTHON=/opt/anaconda2/bin/python

export PYSPARK_PYTHON=/opt/anaconda2/bin/python

/opt/anaconda2/bin/python should be the location of your 2.7 python (this should be same across all clsuter nodes)

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

Re: multiple versions of python issues

wazzu62 — Thu, 13 Sep 2018 04:50:55 GMT

Thanks, this did work for me!

Is there a way to configure the hadoop cluster to use a specific installed version of python?