What's New @ Cloudera

Over the last few quarters, more and more of our customers deployed their production workloads on Cloudera Cloudera AI. Some of them rely on CML with their predictive maintenance use case, others predict churn or detect fraudulent transactions. The common thing between them is that the ml workloads running on CML are critical for their business’s success.   The new CML Backup and Restore capability gives you an extra layer of protection by giving you the ability to resume operations in a timely manner following an outage or crisis. Now, administrators can take on-demand backups of CML workspaces before cluster operations like an upgrade and do periodic backups during off-peak hours. Backed-up CML workspaces can be restored into a new CML workspace in the same or a new CDP environment, and all project artifacts like deployed models and applications will be recovered.  The Backup and Restore capabilities are available on AWS today, and we are planning to roll out

Key features for this releaseAirflow stability and performance. With this release we now use the latest stable Airflow release 2.2.5.  In conjunction with service enhancements, our testing indicates improved stability under higher DAG and task loads. This allows DAG loads to reach up to 1000 per VC, combined with a concurrency of 200-300 parallel Airflow tasks within the same Service. Pre-loaded sample jobs and data for new users To help new users self-learn CDE, all new Virtual Clusters will have the option to load example Airflow & Spark jobs accompanied with sample data.Spark 3 support for raw scala code Previously this feature was limited to Spark 2, it is now extended to Spark 3 based Virtual Clusters.   This allows users to directly run raw scala via API & CLI in batch-mode without having to compile, similar to what spark-shell supportsPipeline UI editor for Airflow is now GA with support for all major browsers (Firefox, Chrome, and Safari). New Virtua

Data Scientists have access to a wide range of ML Runtimes, they can use different versions of Python, R, and Scala kernels with the Workbench or the JupyterLab editors and can benefit from extended capabilities like GPU acceleration. Cloudera continues to release new versions of ML Runtimes, while customers can also register custom ones they build to solve their specific business use case. With the continuous additions, the available options can grow large, and some will become irrelevant or outdated.  CML now supports administrators to disable Runtime variants or specific versions. For example, they can decide to disable all of the Python 3.6 Runtimes as the python kernel is officially EOL and won't receive any further security and bugfix patches. Usage of these runtimes can be considered a security risk that administrators now can solve. Once administrators disable an ML Runtime, data scientists won't be able to use them for development, and existing workloads configured w

The CDP Public Cloud Release Summary for the month of June is now available. It summarizes all major features introduced in CDP Public Cloud Management Console, Data Hub, and data services. See CDP Public Cloud Release Summary - June 2022. 

The latest release (2.1.0-b123) of Cloudera DataFlow (CDF) on Cloudera on cloud introduces the following new features for both, AWS and Azure customers: [Technical Preview] CDF service and deployment upgrades are now supported.CDF service upgrades require a specific entitlement that is granted by request only. Reach out to your Cloudera Account Representative or open a support ticket to request CDF service upgrades.The create-deployment CLI command now supports Inbound Connections.CDF now correctly configures the deployment when it detects any of the two reserved Controller Services Default NiFi SSL Context Service and Inbound SSL Context Service regardless of whether they are external references or not.Kubernetes Version 1.22 is now the default Kubernetes version for CDF. When CDF is enabled, it creates AWS EKS and Azure AKS clusters based on version 1.22.CDF now supports Data Lake scaling. After the Data Lake scale operation has completed, affected

Certain countries may have regulatory requirements that limit or forbid specific types of data from being sent or stored outside of the country or region where an organization operates. For this reason Cloudera has introduced new regions in which the Control Plane can run. By choosing a region other than us-west-1, you can ensure that sensitive metadata (such as information about CDP users) does not leave its region or country of origin. For more information, see CDP Control Plane regions. This is available starting in COD v1.22 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

COD ensures high availability and fault tolerance using Multi-AZ deployments. A Multi-AZ deployment means that compute infrastructure for HBase’s master and region servers are distributed across multiple AZs ensuring that when a single availability zone has an outage, only a portion of Region Servers is impacted and clients automatically switch over to the remaining servers in the available AZs.Multi-AZ for COD is currently supported only on Amazon Web Services (AWS) environments. For more information, see Multi-AZ deployment on COD. This is available starting in COD v1.22 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

Store File Tracking (SFT) defines a separate, plugable layer to handle storefile life cycle, and includes the FILE based built-in implementation that avoids internal file rename or move operations while managing the storefiles. This is a critical enablement to deploy HBase over S3 object store, which is known for the lack of atomic renames. COD enables this feature by default for databases deployed on AWS with S3, to mitigate the aforementioned S3 limitation that could cause critical issues for HBase. For more information, see HBase Store File Tracking. This is available starting in COD v1.22 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

Data scientists on CML Workspaces have access to GPUs to accelerate their machine learning projects and reduce the time it takes to build and train predictive models. NVIDIA GPU nodes are available for administrators to configure for CML Workspaces in both AWS and Azure. CML now supports adding new GPU nodes to existing CML Workspaces created without GPUs, so data scientists can access GPU acceleration without having to recreate CML Workspaces. Administrators can also replace GPU nodes in CML Workspaces to switch to the latest generation GPUs.With these new capabilities, it's easier for administrators to manage GPU nodes in CML Workspaces and enable data scientists to use the newest generation of GPUs. 

 We are pleased to announce the release of Cloudera ODBC Connector 2.6.16 for Impala. This release addresses the following: Enhancements and New Features[IMP-899][00370402] Improvements to idempotent HTTP operations[IMP-904][00371927] New mechanism for SAML SSO[IMP-857] Updated OpenSSL support Resolved Issues[IMP-889] In some cases, when executing a complex query that involves multiple joins, aggregations, and searched case statements, the connector terminates unexpectedly.[IMP-902][00368911] In some cases, when using HTTP, SSL, or proxy connections, the connector returns an "SSL_connect: wrong version number” error. Known Issues[IMP-843][00294390] Limited support for query translation  Getting Started with the Cloudera DriverRead the Cloudera ODBC Connector 2.6.16 for Impala Release Notes and Installation GuideDownload the connector from the Cloudera Connectors page As always, we welcome your feedback. Please send your comments and suggestions t

The Data Discovery and Visualization experience ships with preconfigured Data Connections, a database browser, interactive SQL editor, drag-and-drop Visual Dashboarding, and Connection Snippets. These new capabilities speed up the development process by cutting down the time spent finding, exploring, understanding, and accessing the data. Data Scientists need to fully understand their data in order to analyze it properly, build models, and power ML use cases. To reduce the time to insights, CML ships all tools required to integrate these tools to reduce the friction between the different steps and to speed up the development process for data science teams.  These new capabilities are built on top of Cloudera Data Visualization, giving state-of-the-art visual capabilities at the hand of Data Scientists. To get started, you can step into any Project in a CML May or newer Workspace and hit the Data tab. You can read more about the new capabilities in the documentation

COD now supports replication for the META regions, which means clients can read the META replicas first before connecting to the primary region server. This reduces the load on the META table because the read META load is distributed across multiple replicas.COD supports configuring three META replicas which can be deployed in different RegionServers. Client needs to use the COD provided configuration to get the META calls that are distributed across these replicas.This would be useful in the case where the application has lots of clients like while accessing using spark, yarn, phoenix-spark etc. This is available starting in COD v1.21 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

The CDP Public Cloud Release Summary for the month of May is now available. It summarizes all major features introduced in CDP Public Cloud Management Console, Data Hub, and data services.   See CDP Public Cloud Release Summary - May 2022.  Reply  

The ML Runtimes 2022.04-1 Release includes a technical preview version of the new workbench architecture, the PBJ (Powered by Jupyter) Workbench. In the previous Workbench editor, the code and output shown in the console (the right-hand pane in the image below) were passed to and from Python, R, or Scala via a Cloudera-specific, custom messaging protocol. In the PBJ Workbench, on the other hand, the code and output are now passed to and from the target language runtime via the Jupyter messaging protocol. They are handled inside the runtime container by a Jupyter kernel and rendered in your browser by JupyterLab’s client code. This may seem like a subtle change, but it will provide CML users with some major benefits. First, the behavior of user code and third-party libraries on CML will be more consistent with its behavior in Jupyter-based environments. That means that a wider variety of rich visualization libraries will work out of the box, and in cases where rich visualization li

With the launch of Cloudera on cloud 7.2.15, Cloudera Streaming Analytics for Data Hub deployments has gotten some powerful new features with support for Dead Letter Queues. SQL Stream Builder now has a de-serialization tab that allows users to configure how errors with messages are handled as part of their SQL Queries. Connections using Kafka can setup multiple options for what to do on a failure to process the message rather than causing the job to stop the message can be handled in multiple ways such as: Failing the Job, Ignoring the Message, Ignoring and Logging the Message, ignoring the message and write it to a Dead Letter Queue (another Kafka Topic.)Read more about the de-serialization capabilities in the Streaming Analytics 7.2.15 Documentation.

With the launch of Cloudera on cloud 7.2.15, Cloudera Streams Messaging for Data Hub deployments has gotten some powerful new features! Streams Messaging now supports Multi-Availability Zone Deployments enabling High Availability, OAuth2 support for Clients connecting to Kafka Brokers and Schema Registry, Streams Messaging Manager Connect UI Changes, Kafka Connect security features, Debezium CDC Connectors, ability to import Kafka data to Atlas and much more. Streams Messaging High Availability cluster definition and template. You can use the template and definitions to deploy highly available Streams Messaging clusters that leverage multiple availability zones and ensure that functionality is not degraded when a single availability zone has an outageThree new cluster definitions are introduced for Streams Messaging. The new definitions are as follows:Streams Messaging High Availability for AWSStreams Messaging High Availability for Azure (Technical Preview)Streams Messaging

With the launch of Cloudera Streaming Analytics 1.7.0 Parcel, Cloudera Streaming Analytics for Private Cloud deployments has gotten some powerful new features! Reworked Streaming SQL ConsoleThe User Interface (UI) of SQL Stream Builder (SSB), the Streaming SQL Console has been reworked with new design elements. High Availability for SSBYou can use SQL Stream Builder (SSB) with a Load Balancer to distribute tasks over resources in case of a single point of failure. SQL Server CDC Connector supportMS SQL Server Change Data Capture (CDC) connector is added to the list of supported connectors. The SQL Server CDC connector can be used with Flink DDL on the Streaming SQL Console either directly from the SQL Editor or using the DDL template. Configuring retention time or row count for Materialized ViewsYou can configure how data should be retained for Materialized Views based on time or row count.  Learn more about new features in the Streaming Analytics 1.7.0 Do

This release (CDF-PC 2.0.0-b302) introduces the following new features for both, AWS and Azure customers: Flow Deployments can now be configured with Inbound Connections providing a stable hostname, automated certificate/key generation as well as automated load balancer provisioning to enable client applications to securely send data to a NiFi Flow Deployment.Flow Deployments now support the latest Apache NiFi 1.16 release.Flow Deployments on Azure now use Premium SSD disks to store NiFi’s various repositories.Python 3.9 is now installed on NiFi containers allowing users to run python scripts through NiFi processors. Requests and urllib3 libraries are also installed by default.Information level alerts are now displayed in the Dashboard to inform users about an action they should perform for a Flow Deployment without changing the Flow Deployment status to “Concerning”.Deployments can now be restarted from the Deployment Manager to perform actions that require a NiFi

The CDP Public Cloud Release Summary for the month of April is now available. It summarizes all major features introduced in CDP Public Cloud Management Console, Data Hub, and data services. See CDP Public Cloud Release Summary - April 2022. 

Data warehouse users require constantly improving performance, regardless of data volumes or number of end users, with tools that are ever easier to use. Cloudera just released a capability that answers each of those calls, providing significant performance improvements that apply regardless of scale and that are almost trivial to set up. We are pleased to announce the general availability of Unified Analytics within Cloudera Data Warehouse (CDW) - our cloud native data warehouse service available in Cloudera Data Platform (CDP).   Unified Analytics (UA) boosts performance by orders of magnitude for BI users via intelligent materializations that are applied transparently. This includes materialized views, results caching, and approximate algorithms. It further increases performance by enhancing our query optimizer,  which especially benefits the complex analytic queries that are increasingly common in lakehouse style architectures. Lastly it simplifies development for the dat

Cloudera Logging is a modified Apache Log4j 1.2.x compatible logging library included with Cloudera on premises Base 7.1.7 SP1 that is created, distributed, and maintained by Cloudera to address the recent vulnerabilities in Apache Log4j 1.2.x.While Apache Log4j 2.x is actively maintained by the Apache Software Foundation community, 1.2.x isn't and many Cloudera components rely on Log4j 1.2.x libraries. Because there is limited compatibility between 1.2.x and 2.x, we created Cloudera Logging which was forked from Apache Log4J 1.2.17 and maintained by Cloudera internally to provide customers with a more secure, stable, and maintained logging library that's compatible with 1.2.x and includes security fixes from the Log4j and Reload4j community fork. To help ensure that Cloudera Logging stays current with the latest community work, Cloudera's product security and compliance teams are monitoring the Log4j and Reload4j communities for new issues and, when identified, work to include fixes i

 We are pleased to announce the release of Cloudera JDBC 2.6.27 driver for Impala.  This release has the following enhancements: Enhancements and New Features[IMPJ-673] Updated log4j third-party libraries to 2.17.1[IMPJ-688] Updated Jackson library to 2.13.1 Resolved Issues[IMPJ-691] When close() is not called on the connection object, the connector prevents the JVM from exiting.  Getting Started with the Cloudera DriverRead the Cloudera JDBC Connector 2.6.27 for Impala Release Notes and Installation GuideDownload the connector from the Cloudera Connectors page As always, we welcome your feedback. Please send your comments and suggestions to the user group through our community forums. You can also file bugs through our external Jira projects on issues.cloudera.org.

 We are pleased to announce the release of Cloudera JDBC 2.6.19 driver for Apache Hive.This release has the following fixes and enhancements: Enhancements and New Features[HIVEJ-722] Removed slf4j-log4j12 dependency[00374597][HIVEJ-723] Updated log4j third-party libraries[HIVEJ-725] Updated Jackson library  Getting Started with the Cloudera DriverRead the Cloudera Hive JDBC 2.6.19 Driver for  Apache Hive Release Notes and Installation GuideDownload the connector from the Cloudera Connectors page As always, we welcome your feedback. Please send your comments and suggestions to the user group through our community forums. You can also file bugs through our external Jira projects on issues.cloudera.org.

Cloudera is focused on helping our customers by swiftly innovating on the product and technology side. We accomplish this by bringing powerful new capabilities to our platform that enable customers to drive even greater business value. Cloudera has been providing regular upgrades for customers from our legacy releases on HDP/CDH, and we now designate certain releases as Long-Term Support (LTS) releases to help customers feel confident in upgrading large, complex clusters that require significant planning to upgrade. This is helpful for those who typically do not perform frequent major upgrades and thus depend on the long-term support these releases provide with critical bug fixes, security fixes and a stable platform.   Since the launch of Cloudera on premises Base, we’ve provided 6 new releases of the platform, each introducing innovative new features, platform enhancements, robust upgrade paths for legacy platforms, broader integrations, and of course, bug and security

COD now provides an enhanced user interface to download the Phoenix Client Jar from the Phoenix Thick and Phoenix Thin client tabs through a single click. This is available starting in COD v1.20 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).