Created on 03-24-202002:46 AM - edited on 03-30-202011:43 PM by VidyaSargur
Running custom applications in CDSW/CML
With the recent addition of Analytical Applications, data scientists and data engineers can now deploy their own custom applications and share them with other users.
Whilst simple applications may have all necessary code already baked into one script file, more complicated applications may require custom launchers, run flags that are set on execution and parameters that may be instance specific. In order to run these applications, we can leverage the Python subprocess module to run the commands that we would normally have manually entered into the terminal.
For this demonstration, I will show how to run Tensorboard and Hiplot, where both allow for the visualization of parameters from multiple runs of deep learning models.
Both applications rely on a custom command to trigger them as standalone applications: tensorboard for Tensorboard and hiplot for Hiplot.
CDSW 1.7.1 or later versions
Python 3 engines with web access available (for installing libraries)
I save this out in the run-hiplot.py script. The os.environ["CDSW_APP_PORT] command calls the environment variable CDSW_APP_PORT which specifies which port the application must use in order to run successfully. For Tensorboard:
Notice that adding a flag is as simple as adding the flag and its settings, as a part of the comma separated list within the subprocess.call([ ... ]) command.
For this demonstration, I will generate some data to populate tensorboard first.
Run test_runs_tensorflow.py in the CDSW/CML Session by opening the py file, then click the play arrow in the coding window.
Running applications that require flags
Now that we have the script, we can go ahead and trigger it as an application:
Go the applications screen:
Click New Application:
Fill in the form: Note: The option to Set Environment Variablesjust before the Create Application button. Leveraging os.environ[''] and the ability to set environment variables from the New Applicationscreen, it is still possible to edit run flags without editing the run script.
To access the application, click the box with the arrow:
The new Analytical Applications function rolled out in CDSW 1.7.x and available in CML - Public Cloud enables the deployment of third-party and custom applications on Cloudera Machine Learning infrastructure.
Through the use of the Python subprocess module, it is possible to execute arbitrary code and set runtime flags for applications as well.