What's New @ Cloudera

COD introduces a CldrCopyTable utility, which is Cloudera’s version of the upstream CopyTable utility. This utility is an extension of the COD replication plugin that enables you to copy data cross-realm. This utility enables you to close any replication gaps that might arise depending on how you start or stop replication. This is available starting in COD v1.13 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

The CDP Public Cloud Release Summary for the month of December is now available. It summarizes all major features introduced in CDP Public Cloud Management Console, Data Hub, and data services.   See Public Cloud Release Summary - December 2022  

 We are pleased to announce the release of Cloudera ODBC Connector 2.6.17.1026 for Impala. This release addresses the following: Enhancements and New Features[IMP-927] Updated SSOIgnoreDriverNoPrompt default value[IMP-921][00395874] SAML SSO cookie cache support[IMP-857] Updated OpenSSL support Resolved Issues[IMP- 924][00394280] When the Delegation UID is set, the server parameters are not effected[IMP- 892][00356762] The connector returns successfully during SQLPrepare but throws an error, "Unsupported query" during SQLExecute for some queries Getting Started with the Cloudera DriverRead the ODBC Connector 2.6.17 for Apache Impala Release Notes and Installation GuideDownload the connector from the Cloudera Connectors page As always, we welcome your feedback. Please send your comments and suggestions to the user group through our community forums. You can also file bugs through our external Jira projects on issues.cloudera.org.

You can find the release notes and the download links in the documentation.Key features for this releaseRebase against NiFi 1.18 bringing the latest and greatest of Apache NiFi. It contains a ton of improvements and new features.Reset of the end of life policy: CFM 2.1.5 will be supported until August 2025 to match the CDP 7.1.7 LTS policy. It is particularly important as HDF and CFM 1.x are near to end of life.Parameter Providers: we are introducing the concept of Parameter Providers allowing users to fetch the values of parameters from external locations. In addition to a better separation of duties, it is also very useful to make CI/CD better and easier. With this release, we're supporting the following Parameter Providers:AWS Secret ManagerGCP Secret ManagerHashiCorp VaultDatabaseEnvironment VariablesExternal fileRegistry Client to connect to a DataFlow Catalog. The registry endpoint is now an extension in NiFi. It means that it is no longer limited to accessing a NiFi Registry ins

The latest release (2.3.0-b347) of Cloudera DataFlow (CDF) on Cloudera on cloud introduces the following new features for both, AWS and Azure customers: Flow Designer [Technical Preview]Developers can now build new data flows from scratch using the integrated Designer. Flow Drafts can be tested using Test Sessions before they are published to the Catalog. Once they are in the catalog, users can create Flow Deployments using the Deployment Wizard or CLI.Note:The Flow Designer is a Technical Preview feature and requires an entitlement to be activated. Reach out to your Cloudera Account team to get Technical Preview access. Added support for certificate renewal of inbound connection endpointsSix new ReadyFlows have been added to the ReadyFlow Gallery:Airtable to S3/ADLSBox to S3/ADLSDropbox to S3/ADLSGoogle Drive to S3/ADLSMQTT to KafkaSalesforce to S3/ADLSDataFlow is now also available in the EU and APAC regional CDP control planes.Updated to support using kubernetes 1.23 EKS a

The ML Runtimes 2022.11-1 Release includes the GA version of the new workbench architecture, the PBJ (Powered by Jupyter) Workbench. In the previous Workbench editor, the code and output shown in the console (the right-hand pane in the image below) were passed to and from Python, R, or Scala via a Cloudera-specific, custom messaging protocol. In the PBJ Workbench, on the other hand, the code and output are now passed to and from the target language runtime via the Jupyter messaging protocol. They are handled inside the runtime container by a Jupyter kernel and rendered in your browser by JupyterLab’s client code. This may seem like a subtle change, but it will provide CML users with some major benefits. First, the behavior of user code and third-party libraries on CML will be more consistent with its behavior in Jupyter-based environments. That means that a wider variety of rich visualization libraries will work out of the box, and in cases where rich visualiza

The CDP Public Cloud Release Summary for the month of November is now available. It summarizes all major features introduced in CDP Public Cloud Management Console, Data Hub, and data services.  See CDP Public Cloud Release Summary - November 2022

We're pleased to announce the release of Cloudera Private Cloud (PVC) Data Services 1.4.1, which delivers key new features and improvements across the platform for better enterprise readiness and BI/AI outcomes.  This release appeals to both data practitioners and platform administrators with the following key highlights: Significant new functionality in Data Services such as Quota Management (CDE), support for Spark History Server (CDE), use of deterministic namespace in Kerberos principals (CDW), ability to use CDP CLI to install and manage CDW Private Cloud (CDW). Promoting enterprise readiness with new capabilities such as External Docker Registry, rolling restart of ECS nodes during upgrades, fixes for Critical CVEs, and additional platform certifications.  Please see the Release Notes for CDP PVC Data Services 1.4.1 for the full list of features delivered in this release.  Cloudera’s Professional Services team is available to ensure yo

CML Experiments have been rebuilt, leveraging the MLflow ecosystem to complement CML’s existing strengths in model development and deployment. CML now ships the mlflow SDK and an integrated visual experience that enables experiment tracking and comparison via flexible visuals.  Model development is an iterative process that involves many “experiments” to determine the right combination of data, algorithm, and hyperparameters to maximize accuracy. Multiple versions of the “model” are created as the iterative process continues after the initial deployment. To efficiently develop models while addressing the technical and governance needs for traceability and repeatability, data scientists need to be able to track these different experiments and model versions and see which experiments produced which models. CML Experiment Tracking through MLflow API CML’s experiment tracking features allow you to use the MLflow client library for logging parameters, code versions, metrics,

The CDP Public Cloud Release Summary for the month of October is now available. It summarizes all major features introduced in CDP Public Cloud Management Console, Data Hub, and data services.  See CDP Public Cloud Release Summary - October 2022.

COD now supports the CDP CLI commands update-hbase-configuration and describe-hbase-configuration to update and retrieve the HBase configuration values.For more information, see update-hbase-configuration and describe-hbase-configuration. This is available starting in COD v1.24 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

For Multi-AZ deployments, COD assigns the nodes to multiple regions when multiple strong meta servers exist in the deployment. This is available starting in COD v1.24 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

Cloudera Manager and monitoring systems require more resources than a regular deployment for big clusters with a large number of nodes. To support this, now COD supports a fast gp2 storage when you choose a gateway node of HEAVY type. This is available starting in COD v1.24 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

The CDP Public Cloud Release Summary for the month of September is now available. It summarizes all major features introduced in CDP Public Cloud Management Console, Data Hub, and data services. See CDP Public Cloud Release Summary - September 2022. 

As businesses continue to adopt and build open lakehouses built with Apache Iceberg on CDP, data scientists need easy access to these new table formats, so they don’t spend their time figuring out connection dependencies and configurations. Cloudera Cloudera AI’s Data Connection and Snippet support simplify data access in CDP. Data scientists can use the cml.data library to gain access to a Data Lake via Spark or query their Virtual Warehouse with Hive or Impala. With recent improvements to the cml.data library, CML Snippets now fully support the Iceberg table format for all Spark, Hive, and Impala data connections. To learn more about Data Connection and Snippet read the following article:https://blog.cloudera.com/one-line-away-from-your-data/

Cloudera Cloudera AI now provides a built-in dashboard for monitoring technical metrics relating to deployed CML Models, such as request throughput, latency, and resource consumption. When machine learning models are deployed in production, it’s essential to know whether the model is successfully providing responses to all queries within the required timeframe and to be able to investigate and find the root cause for any failed responses or other downstream issues. Further, it can be challenging to know ahead of time the resource requirements for the model, such as the number of replicas, and amount of memory and CPU allocated to each replica. This can make it difficult to find the balance between the risk of underprovisioning resources leading to slow responses or timed out requests, and the risk of unnecessarily wasting resources that could be used for other workloads.   The new monitoring features for CML Models provide observability that makes these challenges m

 We are pleased to announce the release of Cloudera JDBC 2.6.21 driver for Apache Hive.This release has the following fixes and enhancements: Enhancements and New Features[00371927][HIVEJ-733] Updated SAML SSO authentication behavior Resolved Issues[00391275][HIVEJ-753] After closing HTTP connections, the connector does not clean up native threads Getting Started with the Cloudera DriverRead the Cloudera Hive JDBC 2.6.21 Driver for  Apache Hive Release Notes and Installation GuideDownload the connector from the Cloudera Connectors page As always, we welcome your feedback. Please send your comments and suggestions to the user group through our community forums. You can also file bugs through our external Jira projects on issues.cloudera.org.

 We are pleased to announce the release of Cloudera ODBC 2.6.13 driver for Apache Hive.This release has the following fixes and enhancements: Enhancements and New Features[HAR-1225] [HAR-1227][HAR-1228] Updated platform support[00371927] [HAR-1288] New mechanism for SAML SSO[00374764] [HAR-1289] Override SQL_ATTR_QUERY_TIMEOUT[HAR-1294] Updated OpenSSL library[HAR-1302] Heartbeat messages[HAR-1303] Configure query translation for CTAS syntax[HAR-1305] Configuring the DSN file Resolved Issues[HAR-1220] When using a new TCP connection to make a HTTP request, the connector does not honor cookies[HAR-1287] The HTTP connection client does not reinitialize after an SSL_read error[HAR-1300] In some cases, when using Linux, the connector terminates unexpectedly when resolving host names[HAR-1301] When using UnixODBC on Linux, error messages are truncated[HAR-1304] The connector does not allow the use of server and intermediate certificates that do not have a CRL distributio

CDP SCIM Support for Azure Active DirectoryThe identity team at Cloudera has been working to add the System for Cross-domain Identity Management (SCIM) support to Cloudera Data Platform (CDP) and we're happy to announce the general availability of SCIM on Azure Active Directory! If you know what SCIM is and you want to start using it then here's a link to our docs to get started. Otherwise, read on. It might just save you a bunch of time and hassle. This blog is broken into two parts:Part One (this blog): CDP SCIM Support for Active Directory, which discusses the core elements of CDP’s SCIM support for Azure AD and the pain points that SCIM solves, and Part Two (now up) : SCIM: An Introduction, for readers who might not be familiar with the SCIM standard.  SCIM support on CDP helps ease and, in many cases, resolve the pain points of our customers. Some of these pain points are listed below.  Pain point: user management as code The main win is that we'

We are pleased to announce the release of Cloudera JDBC Connector 2.6.29 for Impala.  This release has the following enhancements and fixes: Enhancements and New Features[00371927][IMPJ-697] Updated SAML SSO authentication behaviour Resolved Issues[00391275][IMPJ-713] After closing HTTP connections, the connector does not clean up native threads Getting Started with the Cloudera DriverRead the Cloudera JDBC Connector 2.6.29 for Impala Release Notes and Installation GuideDownload the connector from the Cloudera Connectors page As always, we welcome your feedback. Please send your comments and suggestions to the user group through our community forums. You can also file bugs through our external Jira projects on issues.cloudera.org.

The latest release (2.2.0-b194) of Cloudera DataFlow (CDF) on Cloudera on cloud introduces the following new features for both, AWS and Azure customers: A streamlined first time user experienceYou can use the new Hello World ReadyFlow to create your first flow deployment without dependencies on source or target systems. New landing pages for the Dashboard and Catalog allow you to get started with just one click.Four additional new ReadyFlows are available in the ReadyFlow GalleryNon-CDP ADLS to S3/ADLSNon-CDP S3 to S3/ADLSListenTCP filter to S3/ADLSListenSyslog filter to S3/ADLSEndpoint Hostnames certificates can now be renewed when using Inbound ConnectionsLearn more about the new features in the CDF-PC documentation and take an interactive tour of CDF-PC.

COD now displays notifications about all the major events on the COD UI. For example, auto-scaling events, auto-healing events, and alerting notifications.The UI notifications is a major step in providing enough information on COD UI so that you do not need to navigate to multiple internal systems used by COD to get the required information as well as be aware of the events happening in the system on an immediate basis. This is available starting in COD v1.23 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

COD allows you to customize the attached storage worker node for an HDFS cluster while creating a database. You can define the EBS volumes for HDFS clusters.You can also view the customized attached storage worker nodes using the ‘describe-database’ and ‘list-database’ commands. This is available starting in COD v1.23 and you do not need to upgrade to a higher version of COD to benefit from this feature. Try out CDP Operational Database for free with Test Drive (no cloud account or setup is needed).

We're pleased to announce the release of Cloudera Private Cloud (PvC) Base 7.1.8 and Cloudera Manager 7.7.1, which introduce key new features to improve the analytics capabilities offered to your business users, enhanced enterprise readiness, as well as additional third party support. This is a cumulative maintenance release that carries forward the features from 7.1.7 Service Pack 1 (SP1) and prior releases.  Here are some highlights of 7.1.8 we’d like to draw your attention to:This release has a concerted focus on platform resilience, delivering vastly improved high availability (HA) of the platform, which rings true to our laser focus on enterprise readiness.  As organizations plan for continuity in the face of an unplanned outage, 7.1.8 promotes HA with Cloudera Manager's support for active-passive HA, replication between Ozone clusters, balanced distribution of data with Ozone, standby NameNodes for added fault-tolerance, Hive ACID table replication for disaster rec

The CDP Public Cloud Release Summary for the month of July is now available. It summarizes all major features introduced in CDP Public Cloud Management Console, Data Hub, and data services. See CDP Public Cloud Release Summary - July 2022.