Reply
New Contributor
Posts: 1
Registered: ‎04-05-2018
Accepted Solution

--deploy-mode option for CDSW

Is there a recommendation of what --deploy-mode (in spark-submit) option should we use from CDSW engines? If we use --deploy-mode client, does that mean the driver program runs in the CDSW engine? 

 

Thanks !

Cloudera Employee
Posts: 28
Registered: ‎07-09-2015

Re: --deploy-mode option for CDSW

Hi,

 

The documentation has an image explaining this: 

https://www.cloudera.com/documentation/data-science-workbench/latest/topics/cdsw_dist_comp_with_Spar...

 

The answer is yes, if you start a Python 2 session and you create a SparkSession object there you will run the Spark application in client mode and the Spark driver will be inside the CDSW session (docker container). 

This is the primary use-case for CDSW.

 

Regards,

Peter

Announcements