Hi,   We are tracking an issue where the docker daemon is not picking up the proxy configuration. The easiest workaround is to download the 'nvidia/cuda:9.2-devel-ubuntu16.04' image to your local computer and move the image with the docker save; scp; docker load commands to the CDSW host. After doing this the 'nvidia-docker run' should work.   Regards, Peter ... View more

Hi Ibi,   Are you hitting this on the 1.4.0 version? We are tracking a bug related to this (DSE-4293) and documented a workaround:  https://www.cloudera.com/documentation/data-science-workbench/latest/topics/cdsw_known_issues.html#security__tls_ssl   Regards, Peter ... View more

Hi,   This is a known issue for the CDSW 1.3 release, please read the documentation about this: https://www.cloudera.com/documentation/data-science-workbench/1-3-x/topics/cdsw_known_issues.html#cds__py4j   I also see that you are trying to create a SparkContext object which still should work but you might be better off using the new Spark 2.x interfaces. You can see a few examples here: https://www.cloudera.com/documentation/data-science-workbench/1-3-x/topics/cdsw_pyspark.html   Regards, Peter ... View more

Hi,   CDSW relies on Kubernetes to allocate system resources to user workloads and Kubernetes doesn't allow sharing a GPU core like it does for CPUs.   The expectations are set in the Kubernetes documentation: Containers (and pods) do not share GPUs. There’s no overcommitting of GPUs. Each container can request one or more GPUs. It is not possible to request a fraction of a GPU. from: https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/   Regards, Peter ... View more

Hi,   The documentation has an image explaining this:  https://www.cloudera.com/documentation/data-science-workbench/latest/topics/cdsw_dist_comp_with_Spark.html   The answer is yes, if you start a Python 2 session and you create a SparkSession object there you will run the Spark application in client mode and the Spark driver will be inside the CDSW session (docker container).  This is the primary use-case for CDSW.   Regards, Peter ... View more

Hello Rob,   You are trying to use the numpy library from inside a map functions iterating over RDDs. The transformation which you specify will run on the Executors which will be hosted on different machines where you have YARN NodeManager running. To make this work you need to make sure that the numpy library is installed on all of the NodeManager machines.   Regards, Peter ... View more

Hello,   I can reproduce the "Engine exited with status 2." if I'm using the v1 engine but it works with the v2 engine.   What version are you using?  - If you are using 1.0.x then I recommend upgrading to 1.1.1. - If you are already using 1.1.1 then you should use the v2 engine. You can go to your Settings menu on the Project page and select the "Base Image v2, docker.repository.cloudera.com/cdsw/engine:2" on the Engines tab. You can also change the default engine in the Admin menu but it will be applied to new projects, for already existing ones you need to select it manually.   Regards, Peter ... View more

Hi,   Did you add the dedicated CDSW host to the cluster in CM?    From the documentation: "Cloudera Data Science Workbench hosts must be added to your CDH cluster as gateway hosts, with gateway roles properly configured." https://www.cloudera.com/documentation/data-science-workbench/latest/topics/cdsw_install.html#configure_edge_hosts   Regards, Peter ... View more