Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Change Python path

Solved Go to solution

Change Python path

New Contributor

I need to change the python that is being used with my CDH5.5.1 cluster. My research pointed me to set PYSPARK_PYTHON in spark-env.sh. I tried that manually without success. I then used Cloudera Manager to set the variable in both the 'Spark Service Environment Advanced Configuration Snippet'  and  'Spark Service Advanced Configuration Snippet' & about everywhere else that referenced spark-env-sh. This hasn't worked and I'm at a lost where to go next.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Change Python path

New Contributor

You need to add the PYSPARK_PYTHON variable to the YARN configuration :
`YARN (MR2 Included) Service Environment Advanced Configuration Snippet (Safety Valve)`
Do that, restart the cluster and you are good to go.

4 REPLIES 4

Re: Change Python path

New Contributor

You need to add the PYSPARK_PYTHON variable to the YARN configuration :
`YARN (MR2 Included) Service Environment Advanced Configuration Snippet (Safety Valve)`
Do that, restart the cluster and you are good to go.

Re: Change Python path

Explorer

Dear SparkeyG, pls would you elaborate on how to add the PYSPARK_PYTHON variable to YARN configuration snippet as per your suggestion? What format do I need to use for the snippet? Would you be so kind and post an example?

Thank you

jaro

Re: Change Python path

New Contributor

From cloudera manager, select Clusters->Spark

Select Configuration

In the search box in the filters box, search for 'Service Environment Advanced'

in the the Spark Service Environment Advanced Configuration Snippet (Safety Valve) box enter something like:

PYSPARK_PYTHON=/path/to/your/python

 

Click Save Changes ; then Distribute the changes and restart your spark cluster

Highlighted

Re: Change Python path

Explorer

thank you, SparkeyG!

 

for the YARN environment snippet should I add the same string (PYSPARK_PYTHON)?