Hello @PabloMO ,

As the Spoiler Error pointed by you,the versions are not matching.

You can check it by running "which python"

You can override the below two configs in /opt/cloudera/parcels/CDH-<version>/lib/spark/conf/spark-env.sh

and restart pyspark.

export PYSPARK_PYTHON=<same version of python>
export PYSPARK_DRIVER_PYTHON=<same version of python>

 Hope it helps.

Thanks & Regards,

Nandini