Good news!   Long story short -- the problem solved and it was not the CDSW/Jupyter specific one 😉   A bit more verbose explanation. For me it looks clearly like Cloudera Manager bug, which should be addressed accordingly.   In the process of investigation, one of my colleagues suggested to check if command-line pyspark shell is working correctly and apparently it wasn't. Checked from edge node, pyspark was able to start, but threw an error message: [user.name@hostname.domain ~]$ pyspark File "/opt/miniconda3/envs/py36/lib/python3.6/site.py", line 177 file=sys.stderr) ^ SyntaxError: invalid syntax Python 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31) Which told me, that Spark took the proper Python 3.6.9 executable from Miniconda installation as expected, but is unable to process its own libraries/modules and that was quite weird... Googling for the error gave me the advice to check if Pathon is not mixing 2.7 and 3.6 environments, in other words, despite reporting that PySpark is started in Python 3.6.9, the error message came from Python 2.7.5 (OS "native" one), so I started to look for spark-conf/spark-env.sh files and found that bottom part of these files is corrupt and looks like: export PYSPARK_DRIVER_PYTHON=/bin/python3 export PYSPARK_PYTHON=/bin/python3export PYSPARK_DRIVER_PYTHON=/bin/python3 export PYSPARK_PYTHON=/bin/python3 Yes, exactly like this, 3 lines instead of 4 and one line concatenated from 2 equal blocks, but without extra new line between them.   So I went to Cloudera Manager -> Spark -> Configuration and found, that both "extra fields", Spark Service Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh and Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-env.sh were containing two lines each:   export PYSPARK_DRIVER_PYTHON=/bin/python3 export PYSPARK_PYTHON=/bin/python3 On the worker nodes, /bin/python3 points to the proper Python 3.6.9 executable in Miniconda3 installation, so it would work fine if Cloudera Manager woud be able to glue both those sections together with just one newline inbetween. But it looks like those both "freetext" fields are simply used "as is", sticked together and glued at the bottom of the spark-env.sh file template. In our particular case it gave pyspark the strange mixture of system's default python 2.7 and conda's 3.6.9. So, at the end of the day we wiped out those "extras" in Cloudera Manager controlled Spark configuration and after restart everything ran smooth as expected.   ... View more