--deploy-mode option for CDSW

[#1]

Is there a recommendation of what --deploy-mode (in spark-submit) option should we use from CDSW engines? If we use --deploy-mode client, does that mean the driver program runs in the CDSW engine? 

Thanks !

[#2 · Accepted solution]

Hi,

The documentation has an image explaining this: 

https://www.cloudera.com/documentation/data-science-workbench/latest/topics/cdsw_dist_comp_with_Spar...

The answer is yes, if you start a Python 2 session and you create a SparkSession object there you will run the Spark application in client mode and the Spark driver will be inside the CDSW session (docker container). 

This is the primary use-case for CDSW.

Regards,

Peter