Apparently, having multiple directories for yarn and yarn logs causes a misconfiguration when writing the yarn-site.xml file. The solution is to go to cloudera manager -> yarn -> Configuration then search for yarn_service_config_safety_valve  Add a new one by pressing the plus sign on the right:   Name: yarn.nodemanager.runtime.linux.docker.default-rw-mounts   Value: /data01/yarn/nm:/data01/yarn/nm,/data02/yarn/nm:/data02/yarn/nm,/data03/yarn/nm:/data03/yarn/nm,/data04/yarn/nm:/data04/yarn/nm,/data01/yarn/container-logs:/data01/yarn/container-logs,/data02/yarn/container-logs:/data02/yarn/container-logs,/data03/yarn/container-logs:/data03/yarn/container-logs,/data04/yarn/container-logs:/data04/yarn/container-logs   Of course you have to specify the directories as what fits your configs.   ... View more

Okay so I solved this problem.   if anyone got something related check on these 3 things: 1- spark version does not mismatch the python version. Spark 2 doesn't support python higher than 3.7. 2- Make sure that your python code starts a spark session, I forgot that I removed that when I was experimenting. 3- Make sure there are no problems in the code it self and test it on another machine to check if it works properly ... View more