Ok, I've fixed it as below and i'm not sure why it's only working when updating the file and not CM I run sudo gedit /opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/spark/conf/spark-env.sh and then added the below line at the very end of the script export PYSPARK_PYTHON=/opt/cloudera/parcels/Anaconda/bin/python running the test python code again and it worked fine ... View more