What's New @ Cloudera

We are happy to announce the latest release of Cloudera Data Engineering for the Public Cloud.

Starting this release, CDE will now support in-place upgrades in Tech Preview (TP) alleviating the need to delete and recreate the entire service.  Please reach out to your account team if you want to participate in the TP program.  Additionally, stability and performance improvements increase the deployment scale of the managed Airflow service.

Starting with the February release (v1.14) of CDE , Apache Iceberg tables are now supported with Spark 3 virtual clusters on AWS. Users can query and work with tables at petabyte scale without impacting query planning, while benefiting from efficient metadata management, snapshotting, and time-travel.

Customers and data practitioners can now use a self-service, point and click interface to author multi-step data pipelines with little to no code required.

Customers with bursty data pipelines with overlapping  schedules can now leverage Cloudera Data Engineering (CDE) new bin-packing auto-scaling policy for more efficient resource allocation and reduce cost.

Cloudera Data Engineering (CDE) now offers the latest version of Airflow as the default managed scheduling service, which brings with it all the new benefits of Airflow 2.1 including: scheduler speedup of up to 17x, task groups, a new-look UI, and a new way of writing DAGs using the TaskFlow API.

Users of Cloudera Data Engineering (CDE) on CDP Public Cloud can now deploy, monitor, and operationalize data pipelines with Spark 3 in addition to Spark 2, within the same environment.

Administrators of Cloudera Data Engineering (CDE) on CDP Public Cloud now have access to real-time as well as historical diagnostic log bundles without the need to access S3 or Kubernetes APIs.

Users of Cloudera Data Engineering (CDE) on CDP Public Cloud can now deploy, monitor, and operationalize data pipelines with Spark 3 in addition to Spark 2, within the same environment.

Users of Cloudera Data Engineering (CDE) on CDP Public Cloud can now deploy, monitor, and operationalize data pipelines on Microsoft Azure at scale.

Supports fault-tolerant access to all cluster web UIs and end-to-end reliability for more workload types.