Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

--deploy-mode option for CDSW

Solved Go to solution

--deploy-mode option for CDSW

New Contributor

Is there a recommendation of what --deploy-mode (in spark-submit) option should we use from CDSW engines? If we use --deploy-mode client, does that mean the driver program runs in the CDSW engine? 

 

Thanks !

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: --deploy-mode option for CDSW

Rising Star

Hi,

 

The documentation has an image explaining this: 

https://www.cloudera.com/documentation/data-science-workbench/latest/topics/cdsw_dist_comp_with_Spar...

 

The answer is yes, if you start a Python 2 session and you create a SparkSession object there you will run the Spark application in client mode and the Spark driver will be inside the CDSW session (docker container). 

This is the primary use-case for CDSW.

 

Regards,

Peter

1 REPLY 1
Highlighted

Re: --deploy-mode option for CDSW

Rising Star

Hi,

 

The documentation has an image explaining this: 

https://www.cloudera.com/documentation/data-science-workbench/latest/topics/cdsw_dist_comp_with_Spar...

 

The answer is yes, if you start a Python 2 session and you create a SparkSession object there you will run the Spark application in client mode and the Spark driver will be inside the CDSW session (docker container). 

This is the primary use-case for CDSW.

 

Regards,

Peter

Don't have an account?
Coming from Hortonworks? Activate your account here