Hello @PabloMO ,
As the Spoiler Error pointed by you,the versions are not matching.
You can check it by running "which python"
You can override the below two configs in /opt/cloudera/parcels/CDH-<version>/lib/spark/conf/spark-env.sh
and restart pyspark.
export PYSPARK_PYTHON=<same version of python> export PYSPARK_DRIVER_PYTHON=<same version of python>
Hope it helps.
Thanks & Regards,