What's New @ Cloudera

With the launch of CDP Public Cloud 7.2.16, Cloudera Streaming Analytics for Data Hub deployments has gotten some powerful new features! In this release, the Streaming Analytics templates in Data Hub will come with Apache Flink 1.15.1, a completely rebuilt SQL Stream Builder UI, support for Software Development Lifecycle projects with direct integrations to GitHub, support for Iceberg data format, Job Notifications, and more.

This release (2.3.0-b347) of Cloudera DataFlow (CDF) on CDP Public Cloud introduces the technical preview of Flow Designer, adds several new ReadyFlows, improves upgrade reliability and fixes issues with stability for clusters with high utilization.

As businesses continue to adopt and build open lakehouses built with Apache Iceberg on CDP, data scientists need easy access to these new table formats, so they don’t spend their time figuring out connection dependencies and configurations.

Cloudera Machine Learning’s Data Connection and Snippet support simplify data access in CDP. Data scientists can use the cml.data library to gain access to a Data Lake via Spark or query their Virtual Warehouse with Hive or Impala. With recent improvements to the cml.data library, CML Snippets now fully support the Iceberg table format for all Spark, Hive, and Impala data connections. 

To learn more about Data Connection and Snippet read the following article:
https://blog.cloudera.com/one-line-away-from-your-data/

Cloudera Machine Learning now provides a built-in dashboard for monitoring technical metrics relating to deployed CML Models, such as request throughput, latency, and resource consumption.

CDP now supports SCIM for Azure Active Directory.

The latest release (2.2.0-b194) of Cloudera DataFlow (CDF) on CDP Public Cloud improves the first time user experience, introduces more ReadyFlows, and supports renewing certificates for Inbound Connection.

CML's Backup and Restore feature is now generally available in the public cloud on AWS. Administrators can backup their CML Workspaces and ensure business continuity in case of failures and outages. 

We are happy to announce the latest release of Cloudera Data Engineering for the Public Cloud.

Starting this release, CDE will now support in-place upgrades in Tech Preview (TP) alleviating the need to delete and recreate the entire service.  Please reach out to your account team if you want to participate in the TP program.  Additionally, stability and performance improvements increase the deployment scale of the managed Airflow service.

The latest release (2.1.0-b123) of Cloudera DataFlow (CDF) on CDP Public Cloud introduces support for DataFlow service upgrades as a technical preview feature, Inbound Connection configuration through the create-deployment CLI command, Kubernetes 1.22 as well as additional improvements and fixes.

With the launch of CDP Public Cloud 7.2.15, Cloudera Streaming Analytics for Data Hub deployments has gotten some powerful new features with support for Dead Letter Queues.

With the launch of CDP Public Cloud 7.2.15, Cloudera Streams Messaging for Data Hub deployments has gotten some powerful new features! Streams Messaging now supports Multi-Availability Zone Deployments enabling High Availability, OAuth2 support for Clients connecting to Kafka Brokers and Schema Registry, Streams Messaging Manager Connect UI Changes, Kafka Connect security features, Debezium CDC Connectors, ability to import Kafka data to Atlas and much more.

The latest release of Cloudera DataFlow for the Public Cloud (CDF-PC 2.0.0-b302) adds support for NiFi flows that listen for incoming data (e.g. through ListenHTTP, ListenTCP, ListenSyslog processors) and allows users to deploy them in a cloud-native, Kubernetes based runtime with improved monitoring and auto-scaling capabilities. 

This release also introduces the latest Apache NiFi 1.16 version as well as additional new features, new ReadyFlows and stability improvements. 

Data warehouse users require constantly improving performance, regardless of data volumes or number of end users, with tools that are ever easier to use. Cloudera just released a capability that answers each of those calls, providing significant performance improvements that apply regardless of scale and that are almost trivial to set up. We are pleased to announce the general availability of Unified Analytics within Cloudera Data Warehouse (CDW) - our cloud native data warehouse service available in Cloudera Data Platform (CDP).

You can now download the Phoenix Client Jars with a single click directly from the Phoenix Thick client and Phoenix Thin client tabs in the UI.

With the launch of CDP Public Cloud 7.2.14, Cloudera Streams Messaging for Data Hub deployments has gotten some powerful new features! In this release, the Streams Messaging templates in Data Hub will come with Apache Kafka 2.8 and Cruise Control 2.5 providing new core features and fixes. KConnect has been added and gains additional capabilities with new connectors and Stateless Apache NiFi capabilities which can run NiFi Flows as connectors.  The Schema Registry will now support JSON schemas in addition to the Apache Avro schemas already supported and will gain the ability to perform native API based import and export to share schemas between environments. 

COD now supports the “Storefile Tracking” (SFT) as an optional feature in Runtime 7.2.14.0.

COD allows to disable the Kerberos authentication temporarily for HBase clients that run on Cloudera legacy products.

COD supports custom table coprocessors, which you can implement and extend from HBase coprocessors’ interfaces.

COD supports RAZ integration from the Runtime version 7.2.11.0. You can grant fine-grained access to directories.

Starting with the February release (v1.14) of CDE , Apache Iceberg tables are now supported with Spark 3 virtual clusters on AWS. Users can query and work with tables at petabyte scale without impacting query planning, while benefiting from efficient metadata management, snapshotting, and time-travel.

The latest release of Cloudera DataFlow for the Public Cloud (CDF-PC 1.1.0-b127) allows Microsoft Azure users to run their Apache NiFi flows in a cloud-native, Kubernetes based runtime with improved monitoring and auto-scaling capabilities. 

This release also introduces the following new features for both, AWS and Azure customers:

• Flow Deployments now support the latest Apache NiFi 1.15 release.
• CDF now creates Kubernetes clusters with version 1.20 in AWS EKS and Azure AKS.
• CDF now supports username/password authentication for AWS environments with non-transparent proxy setups.
• When using the Default NiFi SSL Context Service in a flow deployment, the automatically generated truststore now contains the default cacerts from the local JDK. This ensures that the SSL Context Service can be used with 3rd party applications using certificates from common public certificate authorities.
• Users can now perform the Download Kubeconfig and Manage User Access actions when an enablement request fails. This allows in-depth troubleshooting of a failed enablement attempt.
  
• The following ReadyFlows have been added to the ReadyFlow Gallery:
  ADLS to ADLS Avro ReadyFlow
  Kafka to ADLS Avro ReadyFlow
  
  
  Check out the Announcement blog post and learn more about the new features in the CDF-PC documentation and take an interactive tour of CDF-PC
  

The latest release of Cloudera DataFlow for the Public Cloud (CDF-PC) introduces several new features allowing more Apache NiFi users to run their data flows on CDF-PC.

This includes:

• The ability to deploy NiFi flows which require custom processors / custom controller services
• [Tech Preview] The ability to use CDF-PC on Azure
• Support for AWS setups that use non-transparent proxy servers for outgoing network communication
• Support for private AKS/EKS clusters
• Reduced infrastructure cost through optimized instance use and NiFi deployment sizes
  
  Learn more about the new features in the CDF-PC documentation and take an interactive tour of CDF-PC
  

Scale your Kafka Clusters up and down with a push of a button! Monitoring all your Kafka Cluster Replication flows in a single location! View schema's associated to your Kafka Topics while viewing Atlas Data Lineage! You can do all these things now in CSM 7.2.12.

Customers and data practitioners can now use a self-service, point and click interface to author multi-step data pipelines with little to no code required.

Customers with bursty data pipelines with overlapping  schedules can now leverage Cloudera Data Engineering (CDE) new bin-packing auto-scaling policy for more efficient resource allocation and reduce cost.

Cloudera Data Engineering (CDE) now offers the latest version of Airflow as the default managed scheduling service, which brings with it all the new benefits of Airflow 2.1 including: scheduler speedup of up to 17x, task groups, a new-look UI, and a new way of writing DAGs using the TaskFlow API.

The latest release of Cloudera DataFlow for the Public Cloud (CDF-PC) introduces the ability to manage flow definitions in the catalog and create flow deployments using the CDP Beta CLI. This allows users to fully automate the flow deployment process and integrate it into their CI/CD pipelines.

In addition to new CLI features, this release also introduces the following new features:

• A new ReadyFlow is available to move data between S3 buckets leveraging S3 bucket notifications.

• DataFlow now supports Cloudera RAZ (fine-grained access control) for object store access.

• The Dashboard, Catalog, and Environments pages now persist search queries and applied filters for a better user experience.

• When deploying a flow definition, you can now select whether the NiFi flow should be started automatically.

Learn more about the new features in the CDF-PC documentation and read our detailed Blog Post to automate flow deployments

An upgrade-database option has been provided in CDP CLI beta that enables the upgrade of the operating system to the latest supported version

With the cdpctl command line utility, Cloud Administrators can verify compatibility of existing setups with CDP prior to handing over to the CDP Administrator.  cdpctl now adds support for Microsoft Azure in addition to Amazon Web Services.  

Users of Cloudera Data Engineering (CDE) on CDP Public Cloud can now deploy, monitor, and operationalize data pipelines with Spark 3 in addition to Spark 2, within the same environment.