Announcements

Customers that use CDP Operational Database Experience (or the "Operational Database with SQL" template in DataHub) need to deploy their applications on an edge node that has connectivity with the database.  Edge nodes are pre-configured to work with Cloudera's FreeIPA based DNS solution and within the Kerberos domain of the SDX. Creating an edge node is now as simple as selecting the "COD Edge Node" template in Data Hub and is available for both AWS & Microsoft Azure. 

We are pleased to announce the release of Cloudera JDBC 2.6.15 driver for Apache Hive.This release has the following fixes and enhancements: Enhancements and New Features[00263787][HIVEJ-641] SSO authentication[00289968][HIVEJ-662] Updated SSL support[HIVEJ-665] Simulate transactions[00304256][HIVEJ-670] Updated documentation Known Issues[HIVEJ-577] The JDBC 4.0 version of the driver fails to connect to servers that require encryption using TLS 1.1 or later Workflow ChangesRemoved support for CDH versions 5.11 to 5.16Removed support for JDBC 4 (Java 6) Getting Started with the Cloudera DriverRead the Cloudera Hive JDBC 2.6.15 Driver for  Apache Hive Release Notes and Installation Guide Download the connector from the Cloudera Connectors page As always, we welcome your feedback. Please send your comments and suggestions to the user group through our community forums. You can also file bugs through our external Jira projects on 

RAPIDS Runtimes ship a suite of libraries from NVIDIA that bring the power of accelerated GPUs to standard Data Science operations — be it exploratory data analysis, feature engineering, or model training. The RAPIDS libraries are designed as drop-in replacements for common Python data science libraries like cuDF (pandas), cuPy (numpy), cuML (sklearn) and Dask-CUDA (dask) — enabling GPU acceleration for data science workloads of 5X+ without significant code changes. By leveraging the parallel compute capacity of GPUs the time for complicated data engineering and data science tasks can be dramatically reduced, accelerating the timeframes for Data Scientists to take ideas from concept to production. For more information about RAPIDS see rapids.ai.  Data Scientists now can use the RAPIDS Runtimes that enable end-to-end data science and analytics pipelines entirely on GPUs.To learn more, visit the documentation about the RAPIDS ML Runtimes in CML.

Previously, the settings on a Cloudera Cloudera AI (CML) Workspace created on Microsoft Azure could not be adjusted after the ML Workspace was provisioned. This meant that, for example, the autoscaling range for Azure instances could not be changed in order to provide additional compute to the Users on the ML Workspace. When viewing the details of a given ML Workspace from the CDP Management Console, several settings are now updateable. These include the Allowed Load Balancer Source Ranges, API Server Authorized IP Ranges, and Autoscale Range on the Azure instances. This enables Admins to not only enable the ML Workspace to continue to work even after networking changes, but also either expand or restrict the number of instances that could be made available to handle the various workloads on the ML Workspace.

While Cloudera has documented the prerequisites necessary to provision a new Cloudera Cloudera AI (CML) Workspace in the Public Cloud, steps can be missed which lead to a provisioning failure. Sometimes, these failures do not appear until minutes after a provisioning action has begun, which can lead to frustration in getting CML set up. After an Admin provides the necessary settings and clicks the “Provision Workspace” button, CML will immediately conduct a Validation Check of the provided inputs. Any issues found will be highlighted on the “Provision Workspace” screen, enabling Admins to correct any issues before attempting to provision the ML Workspace again.

Applied ML Prototypes (AMPs) in Cloudera Cloudera AI (CML) enable Users to immediately get started with powerful end-to-end examples of Projects that include all the dependencies, algorithms, and user applications required to directly apply use cases to their own needs. While CML ships with access to a library of AMPs created and maintained by Cloudera’s own Fast Forward Labs team, Admins can now set up customized AMPs that are stored in Azure Repos. When defining the AMP Catalog Sources, Admins have been able to define sources through a Git Repository URL or Catalog File URL, pointing at repositories internal to the organization. Admins can now point these sources to repositories stored in Azure Repos, in addition to other repositories such as GitHub.To learn more, visit the documentation about AMP Catalogs.

ML Runtimes are designed to be the successor to Engines in Cloudera Cloudera AI (CML), moving away from the monolithic architecture of Engines and enabling finer grained control over exactly what is necessary by defining the Editor, Kernel, Edition, and Version needed for the given workload. One core piece previously missing from ML Runtimes was the ability to define Addons that go beyond these settings, in particular the ability to leverage the Spark execution engine through an ML Runtime. Admins can now define Addons for ML Runtimes, including Spark and Hadoop CLI. In doing so, Users can leverage these Addons when running workloads such as Sessions, thereby enabling such functionality as Spark-on-K8s to be run through the ML Runtime.To learn more, visit the documentation about Runtime Addons in CML.

Previously, Admins could assign roles such as “MLUser” at the Environment level, enabling the Users to access the ML Workspaces under that Environment. However, those users would not show up inside the ML Workspace until they log in for the first time, preventing other Users from taking actions such as proactively adding them to Projects. From the Users page, Admins can now sync the list of Users between the CDP Management Console and the given ML Workspace. After doing so, any users who have been assigned access to the ML Workspace through the “MLAdmin”, “MLUser”, or “MLBusinessUser” role would appear appropriately within the ML Workspace. This enables other Users to manage any new Users who have not logged in yet as needed, such as adding them to necessary Teams, Projects, etc.To learn more, visit the documentation about Synchronizing Users in CML.

Business Users, who are meant to use and consume information from Applications deployed in Cloudera Cloudera AI (CML), can now be enabled with a streamlined user experience that not only optimizes their access to the Applications, but also unintrousively limits their ability to access workloads that they do not have access to in CML. Admins can now define Business Users by applying the “MLBusinessUser” role at the Environment level, enabling the Business Users to access the ML Workspaces under that Environment. When viewing a given ML Workspace, Business Users will only see the Applications that have been assigned to them through Projects To learn more, visit the documentation about Business Users in CML.

COD UI has been enhanced to provide the client URL for Python (used in conjunction with the phoenixDB python library) It also provides better failure detection when downloading HBase client configuration files using the provided CURL command.  Instead of failing silently, the command will now emit error messages upon authentication failure

CDP Operational Database (COD) adds the ability to automatically tag underlying compute instances.  This enables customers to leverage negotiated pricing with AWS & Azure where such tags are required.  To use this functionality, create a COD database using the CDP CLI.

We're pleased to announce the release of Cloudera Private Cloud (PVC) Data Services 1.2, which delivers key new features and improvements across the platform, including seamless update capability and Extended Support for OpenShift v4.6.      In this release, we’ve added the following new platform capabilities:   Support for OpenShift 4.6: Deploy CDP on the “Extended Support Update” version of OpenShift, reducing the frequency of required OpenShift upgrades. Resource Groups: Assign the same roles to groups of individuals thereby simplifying management of permissions. In-place Updates: Upgrade Private Cloud and the OpenShift cluster in-place, without disruption, in any order. Custom External Alerts: Connect CDP to an alerts system of choice and receive notifications of any health related issues Custom TLS Certificates: Bring your own TLS certificates for added security, thereby addressing a common request from the financial services sector. Customizing Diag

CDP Operational Database (COD) enhances autoscaling to include considerations in addition to RPC latency.  These result in Optimal performance based on data volume in the databaseImproved database availability when a replicated database faces an outageImproved data management during auto-scale down of the database

Apache Hue now supports querying Apache Phoenix greatly simplifying development for customers.  They can now use Hue to test their queries to ensure correctness before, during and after building their applications as well as run operational analytics on live data.  (See blog)  Apache Hue can be accessed directly from the CDP Operational Database screen and provides a rich set of features including query history, data exploration, etc.

CDP Operational Database (COD) now adds support for complex distributed transactions that can support TPC-C benchmarks out of the box when using it as a scale-out relational database i.e., Apache Phoenix (see blog).  This enables a significant expansion of capabilities for the database enabling customers struggling to manage sharded relational databases to easily use COD as an alternative. There is no configuration required to use this capability.  Simply use the “autocommit(false)” and “commit()” commands in your application code / SQL and any SQL commands within these two commands will be executed atomically just as they would with traditional relational databases.

 We are pleased to announce the release of Cloudera ODBC 2.6.11 driver for Apache Hive.This release has the following fixes and enhancements: Enhancements and New Features[HAR-1180] Updated logging configurations[HAR-1184] Updated OpenSS[HAR-1185] Updated DSN password configuration[00263784][HAR-1191] SSO authentication[HAR-1195] CDP 7.1 support Resolved Issues[00309611][HAR-1214] When reconnecting to a Kerberized cluster using Zookeeper, the driver does not connect Workflow Changes[HAR-1205] Removed support for earlier versions of CDH 5.11 through 5.16 Getting Started with the Cloudera DriverRead the Cloudera Hive ODBC 2.6.11 Driver for  Apache Hive Release Notes and Installation Guide. Download the connector from the Cloudera Connectors page.  As always, we welcome your feedback. Please send your comments and suggestions to the user group through our community forums. You can also file bugs through our externa

The Data Engineering HA template has graduated from Preview status to GA on AWS today, with the release of Cloudera Runtime 7.2.9. It now supports fault-tolerant access to all cluster web UIs and end-to-end reliability for more workload types.  The addition of a load balanced, highly available Apache Knox gateway service configured for transparent failover provides this additional resiliency for end users accessing web interfaces of the cluster and for scheduled or ad hoc workloads.  Specifically, Hive workloads submitted to HiveServer2 via the Knox gateway and workloads that access WebHDFS through Knox can seamlessly tolerate instance failures of master nodes running the Knox gateway.  This capability is thanks to the recent addition of an external load balancer to the cluster, and adds to the highly available HDFS, YARN and HiveServer 2 services offered in earlier versions. Future releases of Data Engineering  HA will support highly available Oozie and C

Cloudera Data Warehouse (CDW) Public Cloud service enables self-serve creation of independent data warehouses and data marts for teams of business analysts without the overhead of bare metal deployments using "CDP Environments". By default, the names of environments are system generated. Customers who want to customize their environment names to comply with internal naming standards can now do so using this Preview feature. CDW now provides a “Custom Cluster ID” check box in the environment activation workflow. This brings the following two benefits.  First, it makes the CDW end points, including jdbc/odbc urls, static as long as the environment is around.  That way, users don’t have to change the jdbc/odbc urls in their BI / ETL tools.  Second, even if the environment is updated and the system generated name changes, the user visible custom cluster id masks the underlying changes.  This way, devops scripts can rely on a static name to manage CDW envi

The Cloudera Data Warehouse service (CDW) now allows you to use an AKS internal load balancer to prevent the pods in your cluster from having public endpoints. This ensures that no clients are able to access your services via the public Internet. To use this feature, you simply select the Enable AKS Internal Load Balancer field in the popup that you use to initially activate the CDW environment. CDW will then take care of the required network configuration behind the scenes. This makes your Kubernetes services accessible only to applications running in the same Azure VNET as the Kubernetes cluster, or to a peered VNET. Applications running in on-premises networks that are connected using Azure ExpressRoute circuits can also access your services. CDW lets you run a more cost effective data warehouse platform, where SLAs are easier to meet. CDW also helps you increase your agility and self-service capabilities so that you can more quickly respond to the needs of the business. B

The Management Console for CDP public cloud improves support for hybrid environments by adding support for CDP Private Cloud Base.  You can now register Private Cloud Base clusters as Classic Cluster, similar to what has been available for CDH and HDP clusters. You do not need inbound connectivity to a cluster running on-premise; the process supports creation of a reverse ssh tunnel if required.  This capability will allow you to use Private Cloud Base as the source for Replication Manager.   This feature has been enabled for all CDP accounts.  To learn how to use this feature, please view the documentation. Support for Data Catalog is also available as a Preview, but Data Catalog and Replication Manager support are mutually exclusive while this feature is in Preview state. Preview features are available by request only. To get this feature enabled in your CDP account, please reach out to your Cloudera sales team. 

The Management Console already provides the capability to generate CLI commands for credential creation (AWS, Azure) as well as registering a new SDX/Data Lake (AWS, Azure).  This greatly simplifies the process of automating environment creation.  With this new feature, this functionality is now extended to Data Hub and makes the process of automating Data Hub cluster creation.   You can either get the command to create a replica of an existing Data Hub cluster, or from a  Data Hub Cluster Definition.   To setup the cdpcli, follow the instructions in our documentation.  A full set of cdpcli instructions is available here.  

The default root volume size for all instances on all cloud providers has been increased to 100GB.  Previously, CDP used different volume sizes depending on the cloud service provider.   You can override this by choosing custom hardware and storage for your data hub clusters by following the instructions in our documentation for Amazon Web Services , Microsoft Azure and Google Cloud.  

A number of new features (Endpoint Access Gateway, Medium Duty SDX) have resulted in CDP exercising a set of AWS APIs that were not used earlier. The default cross account role (available via the CDP Documentation) already includes these APIs and no action is required.  However, customers who are running a custom cross account role policy may need to update their policy to ensure they have added the following actions.  Failure to do so will result in environment creation operations failing. cloudformation:UpdateStackcloudformation:ListStackResourceselasticloadbalancing:DescribeLoadBalancerselasticloadbalancing:DescribeTargetHealthelasticloadbalancing:RegisterTargetselasticloadbalancing:DeregisterTargets For details on how to find the AWS Cross Account role for your environment, please see the documentation on how to change an environment's credential.  For details on finding the Amazon Resource Name of the AWS IAM Role, please see the documentation on modifying

Management Console now allows you to select specific nodes within a node group to include a repair operation.. This will reduce the downtime incurred if only a subset of the nodes are unhealthy.   The CDP Management Console has always provided the ability to repair nodes, but this has been at a host group level. With the recent addition of Medium Duty SDX clusters, this would result in the repair operation happening on both healthy and unhealthy nodes, resulting in a service outage while the repair operation was in progress.   From the Hardware tab of the Data Lake details, you can click the Repair icon to select specific nodes within a host group to repair. For details on what happens during the repair process, please see the documentation on Data Lake Repair.