Cloudera Data Engineering 1.24.1
We are excited to announce the latest release of Cloudera Data Engineering on Cloud. This release introduces advancements focused on optimizing cost, enhancing security, and improving the overall developer and administrative experience.
Key Features
- Suspend and Resume (Technical Preview): Temporarily suspend your entire Cloudera Data Engineering Service during idle periods to achieve up to 80% cost savings.
- Git Integration Support (GA): The Git integration feature is now generally available. Stay tuned for more enhancements in the upcoming releases.
- Security Hardened Images: As part of our ongoing effort to improve security and resilience against Common Vulnerabilities and Exposures (CVEs), new hardened images are used by default during service and virtual cluster creation. In addition, new hardened images for Spark and Airflow will be available that contain runtime components upgrades, including Python and JDK. For more information, please refer to the compatibility matrix.
- Runtime Component Version Exposure on Cloudera Data Engineering UI: To make it easier to identify the component versions, Airflow and Spark runtime versions will now be exposed in the virtual cluster UI as well as job runs.
- Expanded Autoscaling Range Settings on Cloudera Data Engineering UI: Configure autoscaling ranges for both On-demand and Spot instances (for AWS) directly from the UI for Core and All-Purpose Tiers, offering greater control over capacity and costs.
- Apache Airflow Version Upgraded to 2.10.4: The upgrade brings the latest features and improvements for workflow orchestration. Refer to the upstream documentation.
- Kubernetes Version Upgraded to 1.31
Use Cases and Benefits
Reduce Costs with Resource Optimizations
The new Suspend and Resume feature (Technical Preview) allows customers to dramatically reduce compute costs by pausing Cloudera Data Engineering Services during non-operational hours, such as holidays or weekends, without losing any configurations or data. Combined with enhanced autoscaling range settings on the UI, it provides unparalleled flexibility in managing resource consumption and optimizing cloud spending.
Boost Platform Security and Operational Resilience
The introduction of a security-hardened image significantly enhances the platform's defense against vulnerabilities, ensuring a more secure and resilient data engineering environment. This, along with the Airflow and Spark runtime upgrades, provides a more robust and stable foundation for all data engineering workloads.
Streamline Data Engineering User Workflow and Experience
The GA of Git integration enables seamless CI/CD pipelines from local development to production deployment. Furthermore, the exposure of runtime component versions in the UI, coupled with the Apache Airflow 2.10.4 upgrade, gives developers greater transparency and control over their execution environments, simplifying compatibility management and accelerating development cycles.
Ready to Explore?
