Key features for this release
Airflow stability and performance. With this release we now use the latest stable Airflow release 2.2.5. In conjunction with service enhancements, our testing indicates improved stability under higher DAG and task loads. This allows DAG loads to reach up to 1000 per VC, combined with a concurrency of 200-300 parallel Airflow tasks within the same Service.
Pre-loaded sample jobs and data for new users To help new users self-learn CDE, all new Virtual Clusters will have the option to load example Airflow & Spark jobs accompanied with sample data.
Spark 3 support for raw scala code Previously this feature was limited to Spark 2, it is now extended to Spark 3 based Virtual Clusters. This allows users to directly run raw scala via API & CLI in batch-mode without having to compile, similar to what spark-shell supports
Pipeline UI editor for Airflow is now GA with support for all major browsers (Firefox, Chrome, and Safari). New Virtual Clusters will have this feature enabled by default.
Azure Private Storage now supported. New private storage options will appear in the Service creation wizard on Azure.
Editing Virtual Cluster configurations post creation, allows adjusting CPU and memory quotas without having to recreate the VC.
[Technical Preview] In-place upgrades are now supported for CDE services 1.14 and higher for both AWS and Azure. Note: This feature is behind an entitlement.
For more details, please refer to the 1.16 Release notes here.
