Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

--deploy-mode option for CDSW

avatar
New Contributor

Is there a recommendation of what --deploy-mode (in spark-submit) option should we use from CDSW engines? If we use --deploy-mode client, does that mean the driver program runs in the CDSW engine? 

 

Thanks !

1 ACCEPTED SOLUTION

avatar
Super Collaborator

Hi,

 

The documentation has an image explaining this: 

https://www.cloudera.com/documentation/data-science-workbench/latest/topics/cdsw_dist_comp_with_Spar...

 

The answer is yes, if you start a Python 2 session and you create a SparkSession object there you will run the Spark application in client mode and the Spark driver will be inside the CDSW session (docker container). 

This is the primary use-case for CDSW.

 

Regards,

Peter

View solution in original post

1 REPLY 1

avatar
Super Collaborator

Hi,

 

The documentation has an image explaining this: 

https://www.cloudera.com/documentation/data-science-workbench/latest/topics/cdsw_dist_comp_with_Spar...

 

The answer is yes, if you start a Python 2 session and you create a SparkSession object there you will run the Spark application in client mode and the Spark driver will be inside the CDSW session (docker container). 

This is the primary use-case for CDSW.

 

Regards,

Peter