Machine Learning and data science libraries and frameworks have grown at an exponential pace along with algorithmic advancements with the introduction and evolution of Neural Networks to Transformer libraries. To keep up with this innovation, our customers have always asked for a pluggable architecture where the libraries can be chosen and hand-selected by their users, yet works within the ML platform that is enabled by CML from data exploration to model operations. Open source ML Runtimes afford that extensibility to our partners and customers. They can extend and create purpose-built runtimes for data science teams and projects.
With our prior investment in the PBJ (Powered by Jupyter) architecture, we can now rely on open source, community-supported protocols and release a new family of our ML Runtimes to better align with the Jupyter ecosystem. With this rebuilt infrastructure, customers and partners will no longer need to build runtime images starting from Cloudera base images. They will no longer need to restrict themselves to languages and versions that Cloudera has packaged. Any combination of the base image, target language, and language version can be used.
By releasing the PBJ ML Runtimes as open source, we can provide more transparency and detail to our customers regarding the environment they are working in. The Dockerfiles used to build the container images act as detailed documentation for customers to understand their working environment fully. Additionally, the open-sourced ML Runtimes serve as a blueprint to create custom Runtimes, supporting building Runtimes on a custom OS, using a custom kernel, or integrating their existing ML container images with CML.
You can access the first release of PBJ ML Runtimes in our public GitHub repository:
https://github.com/cloudera/ml-runtimes
To learn more about Cloudera ML Runtimes, please visit our documentation.
