You can configure engine CPU and memory configurations in the Admin > Engines tab.  However CDSW does not currently have a quota system, akin to YARN resource pools.  This is a roadmap item we are actively working on.  If you have specific requirements, please let us know.   Tristan ... View more

The current best approach to install system packages is to use the custom engine functionality.  This does require administration access however.   I have also filed a feature request internally to package the Impala ODBC drivers and ensure that the base ODBC drivers are part of the standard base image.   Thanks, Tristan ... View more

The Cloudera Data Science Workbench requires Cloudera Manager 5.13+.  The error you are seeing is due to incompatibilities with earlier version of Cloudera Manager.  Please try upgrading Cloudera Manager and let us know if that doesn't resolve the issue.   Best, Tristan ... View more

Cloudera Data Science Workbench only supports Spark 2.x, it doesn't support the Spark 1.x line. Spark 2 has a number of important improvements for data science workloads which is why we have focused on Spark 2 support only. You can still use Spark 1.x on the same cluster, but only Spark 2 within CDSW. Best, Tristan ... View more

Sorry for the delay providing additional details.   The wildcard DNS needs to work on both the CDSW nodes and on your computer.  For a production installation, you should configure a proper wildcard DNS entry using a domain you control.  You need to control a domain, e.g. company.com, and have proper nameservers (e.g. your internal corporate DNS or something like Route53, GoDaddy, etc).  For instance, if you control the domain company.com and the nameservers point to AWS Route 53, then you should configure the wildcard DNS entries as described in the documention within Route 53.  This will resolve the wildcard entries to your master CDSW node both within AWS and on your computer.   If you're in a corporate environment, typically these types of tasks would be done by a network administrator.   If you are only testing CDSW and do not control a domain, the easiest way is to use a service like xip.io which provides a wildcard DNS automatically pointing to an IP.  However this setup should not be used in production since it is unreliable and delegates your DNS configuration to a third party.   I hope this is helpful.   Best, Tristan ... View more

Hi,   Sparklyr is supported by RStudio, so it may be better to ask this question directory to RStudio or in a forum like StackOverflow.   However looking at the code it appears you are passing a string to copy_to rather than a dataframe.  If  assetstatuses is a dataframe that is available, you can try copying it with copy_to(sc, assetstatuses) without quotes around assetstatuses.   See: http://spark.rstudio.com/reference/sparklyr/latest/copy_to.html   Best, Tristan ... View more

You need to configure a wildcard DNS entry to use CDSW.  See the documentation here:   https://www.cloudera.com/documentation/data-science-workbench/latest/topics/cdsw_install.html#set_up_wildcard_dns   While some features will work without a wildcard, running engines, accessing the terminal, or viewing the Spark UI will not.  Because these URLs are random, you cannot simply add them to your local hosts file.   Thanks, Tristan ... View more

Hi David,   This should be the IP address of the master CDSW node.  We will work to clarify this in the documentation.   Thanks, Tristan ... View more

Hi Pollylaw,   Have you set JAVA_HOME in your the Administration settings?  There is a known issue in 1.0.1 with custom JAVA_HOME directories that can lead to this error message.  A workaround is to remove JAVA_HOME from Admin > Engines and make sure Java is located at a location that can be detected by the bigtop-detect-javahome.sh script.  This will be fixed in a future release.   Please let us know if this workaround does not solve your issue or if this explanation does not match your configuration.   Tristan ... View more

For real installations, you should pull from a repository.  This will ensure all the nodes in your CDSW cluster have access to the image, not just the node where you built it.  Moreover, you should not assume that your Docker image store is persistent across upgrades or in long-running clusters where we may evict less used images to free space. By pushing your custom images to a repository, you will ensure that images are never deleted due to image eviction policies or other administration tasks. ... View more