04-10-2018 01:50 PM
Is there a recommendation of what --deploy-mode (in spark-submit) option should we use from CDSW engines? If we use --deploy-mode client, does that mean the driver program runs in the CDSW engine?
04-11-2018 01:33 AM
The documentation has an image explaining this:
The answer is yes, if you start a Python 2 session and you create a SparkSession object there you will run the Spark application in client mode and the Spark driver will be inside the CDSW session (docker container).
This is the primary use-case for CDSW.