What's New @ Cloudera

Business Users, who are meant to use and consume information from Applications deployed in Cloudera Cloudera AI (CML), can now be enabled with a streamlined user experience that not only optimizes their access to the Applications, but also unintrousively limits their ability to access workloads that they do not have access to in CML. Admins can now define Business Users by applying the “MLBusinessUser” role at the Environment level, enabling the Business Users to access the ML Workspaces under that Environment. When viewing a given ML Workspace, Business Users will only see the Applications that have been assigned to them through Projects To learn more, visit the documentation about Business Users in CML.

COD UI has been enhanced to provide the client URL for Python (used in conjunction with the phoenixDB python library) It also provides better failure detection when downloading HBase client configuration files using the provided CURL command.  Instead of failing silently, the command will now emit error messages upon authentication failure

CDP Operational Database (COD) adds the ability to automatically tag underlying compute instances.  This enables customers to leverage negotiated pricing with AWS & Azure where such tags are required.  To use this functionality, create a COD database using the CDP CLI.

We're pleased to announce the release of Cloudera Private Cloud (PVC) Data Services 1.2, which delivers key new features and improvements across the platform, including seamless update capability and Extended Support for OpenShift v4.6.      In this release, we’ve added the following new platform capabilities:   Support for OpenShift 4.6: Deploy CDP on the “Extended Support Update” version of OpenShift, reducing the frequency of required OpenShift upgrades. Resource Groups: Assign the same roles to groups of individuals thereby simplifying management of permissions. In-place Updates: Upgrade Private Cloud and the OpenShift cluster in-place, without disruption, in any order. Custom External Alerts: Connect CDP to an alerts system of choice and receive notifications of any health related issues Custom TLS Certificates: Bring your own TLS certificates for added security, thereby addressing a common request from the financial services sector. Customizing Diag

CDP Operational Database (COD) enhances autoscaling to include considerations in addition to RPC latency.  These result in Optimal performance based on data volume in the databaseImproved database availability when a replicated database faces an outageImproved data management during auto-scale down of the database

Apache Hue now supports querying Apache Phoenix greatly simplifying development for customers.  They can now use Hue to test their queries to ensure correctness before, during and after building their applications as well as run operational analytics on live data.  (See blog)  Apache Hue can be accessed directly from the CDP Operational Database screen and provides a rich set of features including query history, data exploration, etc.

CDP Operational Database (COD) now adds support for complex distributed transactions that can support TPC-C benchmarks out of the box when using it as a scale-out relational database i.e., Apache Phoenix (see blog).  This enables a significant expansion of capabilities for the database enabling customers struggling to manage sharded relational databases to easily use COD as an alternative. There is no configuration required to use this capability.  Simply use the “autocommit(false)” and “commit()” commands in your application code / SQL and any SQL commands within these two commands will be executed atomically just as they would with traditional relational databases.

 We are pleased to announce the release of Cloudera ODBC 2.6.11 driver for Apache Hive.This release has the following fixes and enhancements: Enhancements and New Features[HAR-1180] Updated logging configurations[HAR-1184] Updated OpenSS[HAR-1185] Updated DSN password configuration[00263784][HAR-1191] SSO authentication[HAR-1195] CDP 7.1 support Resolved Issues[00309611][HAR-1214] When reconnecting to a Kerberized cluster using Zookeeper, the driver does not connect Workflow Changes[HAR-1205] Removed support for earlier versions of CDH 5.11 through 5.16 Getting Started with the Cloudera DriverRead the Cloudera Hive ODBC 2.6.11 Driver for  Apache Hive Release Notes and Installation Guide. Download the connector from the Cloudera Connectors page.  As always, we welcome your feedback. Please send your comments and suggestions to the user group through our community forums. You can also file bugs through our externa

The Data Engineering HA template has graduated from Preview status to GA on AWS today, with the release of Cloudera Runtime 7.2.9. It now supports fault-tolerant access to all cluster web UIs and end-to-end reliability for more workload types.  The addition of a load balanced, highly available Apache Knox gateway service configured for transparent failover provides this additional resiliency for end users accessing web interfaces of the cluster and for scheduled or ad hoc workloads.  Specifically, Hive workloads submitted to HiveServer2 via the Knox gateway and workloads that access WebHDFS through Knox can seamlessly tolerate instance failures of master nodes running the Knox gateway.  This capability is thanks to the recent addition of an external load balancer to the cluster, and adds to the highly available HDFS, YARN and HiveServer 2 services offered in earlier versions. Future releases of Data Engineering  HA will support highly available Oozie and C

Cloudera Data Warehouse (CDW) Public Cloud service enables self-serve creation of independent data warehouses and data marts for teams of business analysts without the overhead of bare metal deployments using "CDP Environments". By default, the names of environments are system generated. Customers who want to customize their environment names to comply with internal naming standards can now do so using this Preview feature. CDW now provides a “Custom Cluster ID” check box in the environment activation workflow. This brings the following two benefits.  First, it makes the CDW end points, including jdbc/odbc urls, static as long as the environment is around.  That way, users don’t have to change the jdbc/odbc urls in their BI / ETL tools.  Second, even if the environment is updated and the system generated name changes, the user visible custom cluster id masks the underlying changes.  This way, devops scripts can rely on a static name to manage CDW envi

The Cloudera Data Warehouse service (CDW) now allows you to use an AKS internal load balancer to prevent the pods in your cluster from having public endpoints. This ensures that no clients are able to access your services via the public Internet. To use this feature, you simply select the Enable AKS Internal Load Balancer field in the popup that you use to initially activate the CDW environment. CDW will then take care of the required network configuration behind the scenes. This makes your Kubernetes services accessible only to applications running in the same Azure VNET as the Kubernetes cluster, or to a peered VNET. Applications running in on-premises networks that are connected using Azure ExpressRoute circuits can also access your services. CDW lets you run a more cost effective data warehouse platform, where SLAs are easier to meet. CDW also helps you increase your agility and self-service capabilities so that you can more quickly respond to the needs of the business. B

The Management Console for CDP public cloud improves support for hybrid environments by adding support for CDP Private Cloud Base.  You can now register Private Cloud Base clusters as Classic Cluster, similar to what has been available for CDH and HDP clusters. You do not need inbound connectivity to a cluster running on-premise; the process supports creation of a reverse ssh tunnel if required.  This capability will allow you to use Private Cloud Base as the source for Replication Manager.   This feature has been enabled for all CDP accounts.  To learn how to use this feature, please view the documentation. Support for Data Catalog is also available as a Preview, but Data Catalog and Replication Manager support are mutually exclusive while this feature is in Preview state. Preview features are available by request only. To get this feature enabled in your CDP account, please reach out to your Cloudera sales team. 

The Management Console already provides the capability to generate CLI commands for credential creation (AWS, Azure) as well as registering a new SDX/Data Lake (AWS, Azure).  This greatly simplifies the process of automating environment creation.  With this new feature, this functionality is now extended to Data Hub and makes the process of automating Data Hub cluster creation.   You can either get the command to create a replica of an existing Data Hub cluster, or from a  Data Hub Cluster Definition.   To setup the cdpcli, follow the instructions in our documentation.  A full set of cdpcli instructions is available here.  

The default root volume size for all instances on all cloud providers has been increased to 100GB.  Previously, CDP used different volume sizes depending on the cloud service provider.   You can override this by choosing custom hardware and storage for your data hub clusters by following the instructions in our documentation for Amazon Web Services , Microsoft Azure and Google Cloud.  

A number of new features (Endpoint Access Gateway, Medium Duty SDX) have resulted in CDP exercising a set of AWS APIs that were not used earlier. The default cross account role (available via the CDP Documentation) already includes these APIs and no action is required.  However, customers who are running a custom cross account role policy may need to update their policy to ensure they have added the following actions.  Failure to do so will result in environment creation operations failing. cloudformation:UpdateStackcloudformation:ListStackResourceselasticloadbalancing:DescribeLoadBalancerselasticloadbalancing:DescribeTargetHealthelasticloadbalancing:RegisterTargetselasticloadbalancing:DeregisterTargets For details on how to find the AWS Cross Account role for your environment, please see the documentation on how to change an environment's credential.  For details on finding the Amazon Resource Name of the AWS IAM Role, please see the documentation on modifying

Management Console now allows you to select specific nodes within a node group to include a repair operation.. This will reduce the downtime incurred if only a subset of the nodes are unhealthy.   The CDP Management Console has always provided the ability to repair nodes, but this has been at a host group level. With the recent addition of Medium Duty SDX clusters, this would result in the repair operation happening on both healthy and unhealthy nodes, resulting in a service outage while the repair operation was in progress.   From the Hardware tab of the Data Lake details, you can click the Repair icon to select specific nodes within a host group to repair. For details on what happens during the repair process, please see the documentation on Data Lake Repair.

You can now run Data Hub clusters in your Google Cloud Platform account.  To make it easier to get started with GCP on Google Cloud, a Quick Start is available that will walk you through the prerequisite set up required in your cloud provider account, as well as creating a CDP credential and registering an environment in CDP. This will greatly reduce the amount of time it takes to set up a Proof of Concept for CDP.  CDP requires a number of pre-requisites to be set up in your GCP Project before an environment can be created.  The Quick Start includes GCP Deployment Manager templates to automate creation of pre-requisite resources including VPC network, a subnet, firewall rules, service accounts and storage buckets.   You can get started by reading the Quick Start on Google Cloud documentation.  Alternatively, if you'd prefer to set things up yourself, you can follow the documentation on working with a Google Cloud environment.&nbsp

New, lightweight and customizable Cloudera ML Runtimes are now available in CDP Machine Learning and Cloudera Data Science Workbench 1.9. ML Runtimes have been rebuilt from the ground up to enable greater flexibility in frameworks, processing, and IDEs — powering customizable lightweight deployments without over encumbering the runtime profile. The new profiles are designed to meet the diverse needs of Data Scientists by enabling a variety of ML Runtimes natively and eliminating sizing and versatility issues with previous Cloudera Engine profiles. ML Runtimes with version 2020-02 offer support for Python 3.6, 3.7, and 3.8 with both the Workbench editor and with JupyterLab (GA). All of these runtimes are offered as NVIDIA GPU edition with out-of-the-box GPU acceleration.New R 3.6 and 4.0 Runtimes are also available with the Workbench editor.  To learn more about our vision and roadmap read the Announcement blog post. As always, we welcome your feedback. Please send y

 We are pleased to announce the release of Cloudera JDBC 2.6.23 driver for Apache Impala.This release has the following fixes and enhancements: Enhancements and New Features[IMPJ-604] SSO authentication - You can now authenticate using single sign-on (SSO) via SAML 2.0 on supported servers. Workflow Changes[IMPJ-536] Removed support for JDBC 4.0 (Java 6) Getting Started with the Cloudera DriverRead the Cloudera Impala JDBC 2.6.20 Driver for  Apache Impala Release Notes and Installation Guide. Download the connector from the Cloudera Connectors page.  As always, we welcome your feedback. Please send your comments and suggestions to the user group through our community forums. You can also file bugs through our external Jira projects on issues.cloudera.org. 

 We are pleased to announce the release of Cloudera ODBC 2.6.13 driver for Apache Impala.  This release has the following enhancements and fixes: Enhancements and New Features[IMP-805][00263785] New authentication method[IMP-782] Updated authentication mechanism support Resolved Issues[IMP-846][00280482] When HTTP is enabled and two API calls with a time lag between them are made, the server resets the connection and the driver returns an error. Known Issues[IMP-843][00294390] The driver does not support query translation if the query contains the "LIMIT x OFFSET y" syntax.  Getting Started with the Cloudera DriverRead the Cloudera Impala ODBC 2.6.12 Driver for  Apache Impala Release Notes and Installation Guide. Download the connector from the Cloudera Connectors page.  As always, we welcome your feedback. Please send your comments and suggestions to the user group through our community forums. You ca

We are excited to announce that CDP Public Cloud Data Hub is now available on Google Cloud. This launch builds on our commitment to delivering an enterprise data cloud that enables our customers to do more with their data in a multi-cloud, hybrid deployment model. By providing the option to deploy CDP on Google Cloud, our customers will now have the flexibility and scale they need on the cloud platform of their choice.With CDP Public Cloud on Google Cloud, Cloudera customers can now use SDX and Data Hub to create secure data lakes in their Google Cloud account and provision Data Hub clusters in a matter of minutes.Today, CDP Public Cloud on Google Cloud delivers Data Hub with Data Engineering (Spark, Hive), Data Flow (NiFi) and Streams Management (Kafka) templates. Additional templates will be available in the coming months. In addition to the built-in cluster templates, customers can create custom cluster templates that combine any of the supported services. The combination of th

 CDS 3.1 powered by Apache Spark 3.1.1 is now generally available for CDP Private Cloud Base 7.1.6.This is a minor release of CDS 3. The main improvements include: The parcel contains spark3 compliant spark-hbase connectorAll performance enhancement of Apache Spark 3.1.1 such as new optimizer rules and improved subexpression eliminationsUnify create table SQL syntaxShuffled hash join improvement It can be installed as an add-on parcel on top of CDP Private Cloud Base 7.1.6. The bits can be found here and the latest documentation can be found here.  For Public Cloud the same bits will be available in a pre-warmed image for Cloudera Runtime 7.2.9. That can be installed through Data Engineering Spark3 Data Hub cluster templates. --------------Want to become a pro Spark user? Sign up for Apache Spark Training. 

We're pleased to announce the release of Cloudera Private Cloud (PvC) Base 7.1.6, which provides additional platform support and introduces key new features.  This is a cumulative maintenance release that carries forward the improvements from CDP PvC Base 7.1.5 and prior releases.   A key focus of this release has been to make migrations to CDP PvC Base easier, in addition to new features, and general platform support.  Here’s an overview of the new features and improvements we’ve added in this release.   Upgrade enhancementsTo help customers with their move to CDP PvC Base, we’ve made the following features available in this release: To support upgrading from HDP 3, we’ve released in-place upgrades for directly migrating your clusters to CDP PVC Base, and released a new version of the AM2CM tool (1.2.0) to support migrating from Ambari to Cloudera Manager.Documented rollback procedures for CDH 5 to CDP 7.1.6 and HDP 2 to CDP 7.1.6 are now availa

We are pleased to announce the release of Cloudera ODBC 2.6.12 driver for Apache Impala.  This release has the following enhancements and fixes: Enhancements and New Features[IMP-782][00189899] Updated support for the HTTP transport mode[IMP-801] Updated logging configurations[IMP-802] Updated OpenSSL support[IMP-780][IMP-819][IMP-821][IMP-825][IMP-827] [IMP-835] Updated platform support (for a list of supported platform versions, see the Installation and Configuration Guide)[IMP-807] Updated CDP support. Resolved Issues[IMP-806][00265194] When the server returns 0 rows, the driver stops fetching results.  Known IssuesThe following are known issues that you may encounter due to limitations in the data source, the driver, or an application. • BINARY is not a supported data type in Impala. • DATE is supported only when you connect to Impala version 3.3 or later. • As a result of the limitations above, some values of TIME data types are also not sup

We are pleased to announce the release of Cloudera JDBC 2.6.20 driver for Apache Impala.This release has the following fixes and enhancements: Enhancements and New Features[IMPJ-601] Updated third-party librariesThe JDBC 4.1 driver has been updated to use the following libraries:log4j 2.2.1slf4j 1.7.30[IMPJ-607] Updated CDP supportThe driver now supports CDP 7.1For a list of supported CDP versions, see the Installation and Configuration Guide Resolved Issues[00265194][IMPJ-606] When the server returns 0 rows, the driver stops fetching results[00276449][IMPJ-610] When executing consecutive statements, the driver does not update the parameter values[00276906][IMPJ-611] The driver incorrectly classifies queries containing comments and whitespaces  Known Issues The following are known issues that you may encounter due to limitations in the data source the driver or an application.The JDBC 4.0 version of the driver fails to connect to servers that require encryption usin