I've go a CHD 5.8 cluster that uses Cloudera Manager and it's got a gateway node that I use to launch applications. I'd like be able to dynamcially start and stop Docker containers that are automatically configured to use the cluster. For example, I might start up a Jupyter or R Studio container for a single user and I want those tools to be integrated with my cluster. Cloudera Manager will allow me to add a gateway node to my configuration, but I want the containers to come up ready to launch jobs on the cluster. I've got a number of ideas of how to do this, but I'm sure other folks are doing this, and I'd like to hear if there are any best practices for this. Thanks in advance.
I'm not at all familiar with R Studio or Jupyter but I'd expect you're just asking how these containers can talk to CDH services like Impala or HDFS to submit jobs and read content. If this is what you're referring to, then it would be easiest to just download the client configuration from Cloudera Manager for each service you want and add them to the container before it is launched. Or set up a directory mapping to the local copies of these files. Does this help at all?