About SparkeyG

SparkeyG · ‎04-21-2016

From cloudera manager, select Clusters->Spark Select Configuration In the search box in the filters box, search for 'Service Environment Advanced' in the the Spark Service Environment Advanced Configuration Snippet (Safety Valve) box enter something like: PYSPARK_PYTHON=/path/to/your/python Click Save Changes ; then Distribute the changes and restart your spark cluster

SparkeyG · ‎03-04-2016

You need to add the PYSPARK_PYTHON variable to the YARN configuration : `YARN (MR2 Included) Service Environment Advanced Configuration Snippet (Safety Valve)` Do that, restart the cluster and you are good to go.

SparkeyG · ‎03-04-2016

I need to change the python that is being used with my CDH5.5.1 cluster. My research pointed me to set PYSPARK_PYTHON in spark-env.sh. I tried that manually without success. I then used Cloudera Manager to set the variable in both the 'Spark Service Environment Advanced Configuration Snippet' and 'Spark Service Advanced Configuration Snippet' & about everywhere else that referenced spark-env-sh. This hasn't worked and I'm at a lost where to go next.

Online	Offline
Last Visited	‎04-05-2017 01:56 PM

Member Since	‎09-18-2015 08:39 AM
Last Visited	‎04-05-2017 01:56 PM
Posts	4
Kudos received	3

Cloudera Community

Re: Change Python path

Re: Change Python path

Re: Change Python path

Change Python path