What's New @ Cloudera

With the release of CDP 7.2.11, you now have the possibility to scale up and down both your light duty and heavy duty Flow Management clusters on all cloud providers. You can find more information in the documentation.

CDP Operational Database expands support for SqlAlchemy APIs with Phoenix with support for the getPrimaryKeys and getIndexInfo from the JDBC API through PhoenixDB (along with the SQL Alchemy wrapper APIs).   getPrimaryKeys API retrieves a description of a table's primary key columns ordered by COLUMN_NAMEgetIndexInfo API retrieves a description of a table's indices ordered by NON_UNIQUE, TYPE, INDEX_NAME, and ORDINAL_POSITION This capability is available in PhoenixDB 1.1.0 and is compatible with all CDP 7.x versions and all CDP Operational Database service versions.

We’re thrilled to announce that all Cloudera on cloud customers now have access to Cloudera DataFlow for the Public Cloud (CDF-PC), the first cloud-native runtime for Apache NiFi data flows.  CDF-PC enables Apache NiFi users to run their existing data flows on a managed, auto-scaling platform. With a streamlined way to deploy NiFi data flows and a central monitoring dashboard, it is now easier than ever before to operate NiFi data flows at scale in the public cloud. CDF-PC provides the following features:Import existing NiFi flows and store them in a central catalogDeploy NiFi flows from the central catalog to auto-scaling Kubernetes clustersAllocate resources and set guardrails for individual NiFi deploymentsDefine and monitor KPIs on key NiFi data flow metricsMonitor all your flow deployments from a central dashboardPush-Button NiFi version upgradesGet started with new data flows by using ReadyFlow templates for the most common data movement patternsConsumption based billin

CDP Operational Database (COD) provides increased visibility into performance with graphs on P99 latency, block cache hit ratio, compaction queue size. P99 get latency is one of the key metrics used to determine auto-scaling behavior in COD.  P99 refers to the maximum latency 99% of the requests incur (i.e., 99% of requests are responded to in less than this latency figure).  In other words, only 1% of the requests are expected to be slower than this metric. Block cache hit ratio refers to the # of requests/minute that are serviced from the block cache instead of object storage.  This is an overall driver of database performance as object storage (S3 on AWS, ABFS on Microsoft Azure) is slow relative to the block cache. Compaction queue size gives an indication of the number of Stores in HBase that have been targeted for compaction.  Try out CDP Operational Database for free with Test Drive (no cloud account or setup needed).

The Impala Virtual Warehouses within Cloudera Data Warehouse (CDW) give users an intelligent way to run fast BI queries. Coordinators are the ones that BI tools, such as HUE or Tableau, connect to. There is either one of these running, or two of them if you choose HA mode. Previously, these nodes were always running so that the BI tool connections would not break, and new queries could always be accepted. Now, CDW is able to terminate Coordinators automatically when all queries stop, and restart them once new queries arrive - all without interfering with BI tool connections. Since Impala Executor nodes already had the ability to auto-suspend, this means that all of the largest (i.e. most expensive) nodes used by the Impala Virtual Warehouse now support auto-suspend, dramatically reducing your CDW cost. This capability is currently released as a Tech Preview feature, so please request the Impala Coordinator Auto-Suspend entitlement from your Cloudera account team if you w

Replication Manager dramatically simplifies setup of CDP Operational Database replication automating most parts of setup including:Deployment of the OpDB Replication plugin which eliminates the need for Kerberos x-realm trustCreation of the /user/hbase directory used for snapshot stagingCreation of the table name on target for replicated tablesConfiguration of HBase to setup peering between source & targetConfiguration of HBase replication to enable replicationCreation of snapshot on source and loading of snapshot on target to ensure all existing data is copied as HBase replication only copies data that has changed after it is enabledAutomated rolling restarts after peering Delivered in COD service version 1.13 (deployed services automatically use the latest release) and Cloudera Runtime 7.2.10.1 (requires DataLake and database upgrade)Try out CDP Operational Database for free with Test Drive (no cloud account or setup needed).

COD can now deploy using HDFS on EBS to provide high-performance storage (single millisecond / low double-digit millisecond latencies for many gets/puts).  Use the CDP CLI Beta to deploy a COD with this capability (e.g., “create-database” with “--use-hdfs” flag) Delivered in COD service version 1.13 (deployed services automatically use the latest release) and requires CDP CLI Beta v 0.9.41Try out CDP Operational Database for free with Test Drive (no cloud account or setup needed).

CDP Operational Database (COD) databases store data in cloud storage. The location of the data is stored in cloud storage is defined by the data access cloud storage location in the CDP environment and the COD database name. When a COD database is dropped, any data stored in the cloud storage is not dropped.  You can delete the data separately if you no longer need it. If a COD database is created using the same name (i.e., against a cloud storage location that already contains COD data from a previously deployed database), COD will automatically re-create the tables with the corresponding data in the new COD instance.  Delivered in COD service version 1.13 (deployed services automatically use the latest release)Try out CDP Operational Database for free with Test Drive (no cloud account or setup needed).

We're pleased to announce the release of Cloudera Private Cloud (PvC) Base 7.1.7, which introduces key new features to improve the management, security, and analytical capabilities of the platform and provides additional platform support. This is a cumulative maintenance release that carries forward the improvements from CDP PvC Base 7.1.6 and prior releases.   In addition to new features and platform support, an important goal of the last few releases has been to make the transition to CDP PvC Base easier by adding more upgrade options, and this release continues to deliver against that goal. Here’s an overview of the new features and improvements we’ve added in this release.   Upgrade enhancementsTo help customers with their move to CDP PvC Base we’ve made the following upgrade enhancements in this release:  In-place upgrades are now available for CDH versions 6.1.x, 6.2.x, and 6.3.x., completing our in-place upgrade vision from all supported C

In Cloudera on cloud, each SDX cluster creates a separate kerberos domain and COD databases deploy within the context of each SDX cluster. Prior to this release, using CopyTable required configuration of kerberos x-realm trust between multiple domains. With this release, the HBase CopyTable can now use the same auth mechanism as the OpDB Replication Plugin to copy data across databases without requiring kerberos x-realm trust. Delivered in Cloudera Runtime 7.2.10.1 (requires DataLake and database upgrade to this runtime version) Try out CDP Operational Database for free with Test Drive (no cloud account or setup needed).

With the cdpctl command line utility, Cloud Administrators can verify compatibility of existing setups with CDP prior to handing over to the CDP Administrator.  This will greatly reduce the probability of errors during onboarding into an existing cloud account.   Cloudera on cloud customers often choose to provision their environments into existing cloud accounts where resources such as networking, IAM and storage are pre-created by IT and vetted for adherence to corporate standards.   Getting this right often takes several iterations between the Cloud Administrator and the CDP Administrator. In larger organizations with well-defined organizational boundaries, this exercise can stretch into days. Cloud Administrators can now download the cdpctl utility and use it to verify compatibility of existing resources with CDP.  This release of cdpctl validates AWS infrastructure against a list of our most common failure scenarios, including verifying networkin

Administrators can customize the Cloudera provided ML Runtimes to support Data Scientists’ specific use-cases. They can install additional OS packages, Python and R libraries, third-party drivers to enable connecting to external data stores, or even a new editor to be used. CML now enables registering these custom Runtimes and making them available for Data Scientists to use in their projects.  Data Scientists have specific requirements for their working environments, they require a set of R or Python libraries and ready-made connections to fetch from third-party data stores. With the new feature, administrators can create custom Runtimes that data scientists can use in CML. To learn more, visit the documentation about Customized ML Runtimes.

Cloudera ML Runtimes are the default and recommended solution for running user workloads. New Projects will be created with ML Runtimes configured by default and we recommend migrating existing Projects to use ML Runtimes. Legacy Engines are deprecated and will be removed in a future release but workloads running on them remain fully supported.  To learn more, visit the documentation about Engine deprecation.

Administrators of Cloudera Data Engineering (CDE) on Cloudera on cloud now have access to real-time as well as historical diagnostic log bundles without the need to access S3 or Kubernetes APIs.  Bundles are streamed directly to the users local machine pre-packaged as zip files that can easily be uploaded to Support tickets.   Previously administrators of CDE had to use kubectl to access service pods during troubleshooting scenarios, requiring privileged access and manual steps.   Now, within each CDE service, two types of bundles are available, providing different metrics about the health of the underlying kubernetes cluster and service applications.  The first bundle is a summary snapshot of the state of active pods.  The second, allows the administrator to choose a time period along with specific service applications (such cluster autoscaler) to extract historical logs from. To learn more visit the documentation.

CDP Operational Database Experience (COD) enables customers to use Apache Spark on the same database either directly with Apache HBase (NoSQL) or through Apache Phoenix (SQL) simultaneously through the Spark-Phoenix and Spark-HBase connector enabling customer's whose aggregation pipelines were built using both HBase and/or Phoenix to use the Cloud without code change.  

Cloudera Data Engineering (CDE) now supports multi-version Spark pipelines.  Users can easily test and promote Spark 2 workloads to Spark 3 to take advantage of the performance and stability improvement in the latest version of Spark.  (Performance improvement of over 30% based on internal TPC-DS benchmarks) Data engineers can run workloads in both Spark 2 and Spark 3 within the same CDP PC environment, therefore maintaining backwards compatibility with legacy workloads while developing new applications on the latest version of Spark.  Administrators have a new option within the Virtual Cluster creation wizard to choose a Spark version.  Once up and running, users can seamlessly transition to deploying their Spark 3 jobs through the same UI and CLI/API as before, with comprehensive monitoring of their pipelines including real-time logs and Spark UI. Future release, will include the visual performance profiler into Spark 3 job run details. To learn mor

Users of Cloudera Data Engineering (CDE) on Cloudera on cloud can now deploy, monitor, and schedule data pipelines on Microsoft Azure. Customers can take advantage of a fully managed Spark-on-Kubernetes with autoscaling compute and guard rails to control cost, without the usual platform management overhead.   As opposed to traditional job submissions mechanisms that require direct access to the cluster via edge nodes, data engineers can deploy data pipelines to autoscaling Virtual Clusters through simple, browser-based UI wizards or a full-fledged CLI and API.  Once deployed, CDE optimizes execution through performance metric profiling and real-time monitoring, and provides users with a comprehensive view of their pipelines.  And for further operationalization, a managed Apache Airflow service can be used to orchestrate complex pipelines on a schedule or based on event triggers. Future releases, will add support for Spot and SSD instances, as well as Private Li

Customers that use CDP Operational Database Experience (or the "Operational Database with SQL" template in DataHub) need to deploy their applications on an edge node that has connectivity with the database.  Edge nodes are pre-configured to work with Cloudera's FreeIPA based DNS solution and within the Kerberos domain of the SDX. Creating an edge node is now as simple as selecting the "COD Edge Node" template in Data Hub and is available for both AWS & Microsoft Azure. 

We are pleased to announce the release of Cloudera JDBC 2.6.15 driver for Apache Hive.This release has the following fixes and enhancements: Enhancements and New Features[00263787][HIVEJ-641] SSO authentication[00289968][HIVEJ-662] Updated SSL support[HIVEJ-665] Simulate transactions[00304256][HIVEJ-670] Updated documentation Known Issues[HIVEJ-577] The JDBC 4.0 version of the driver fails to connect to servers that require encryption using TLS 1.1 or later Workflow ChangesRemoved support for CDH versions 5.11 to 5.16Removed support for JDBC 4 (Java 6) Getting Started with the Cloudera DriverRead the Cloudera Hive JDBC 2.6.15 Driver for  Apache Hive Release Notes and Installation Guide Download the connector from the Cloudera Connectors page As always, we welcome your feedback. Please send your comments and suggestions to the user group through our community forums. You can also file bugs through our external Jira projects on 

RAPIDS Runtimes ship a suite of libraries from NVIDIA that bring the power of accelerated GPUs to standard Data Science operations — be it exploratory data analysis, feature engineering, or model training. The RAPIDS libraries are designed as drop-in replacements for common Python data science libraries like cuDF (pandas), cuPy (numpy), cuML (sklearn) and Dask-CUDA (dask) — enabling GPU acceleration for data science workloads of 5X+ without significant code changes. By leveraging the parallel compute capacity of GPUs the time for complicated data engineering and data science tasks can be dramatically reduced, accelerating the timeframes for Data Scientists to take ideas from concept to production. For more information about RAPIDS see rapids.ai.  Data Scientists now can use the RAPIDS Runtimes that enable end-to-end data science and analytics pipelines entirely on GPUs.To learn more, visit the documentation about the RAPIDS ML Runtimes in CML.

Previously, the settings on a Cloudera Cloudera AI (CML) Workspace created on Microsoft Azure could not be adjusted after the ML Workspace was provisioned. This meant that, for example, the autoscaling range for Azure instances could not be changed in order to provide additional compute to the Users on the ML Workspace. When viewing the details of a given ML Workspace from the CDP Management Console, several settings are now updateable. These include the Allowed Load Balancer Source Ranges, API Server Authorized IP Ranges, and Autoscale Range on the Azure instances. This enables Admins to not only enable the ML Workspace to continue to work even after networking changes, but also either expand or restrict the number of instances that could be made available to handle the various workloads on the ML Workspace.

While Cloudera has documented the prerequisites necessary to provision a new Cloudera Cloudera AI (CML) Workspace in the Public Cloud, steps can be missed which lead to a provisioning failure. Sometimes, these failures do not appear until minutes after a provisioning action has begun, which can lead to frustration in getting CML set up. After an Admin provides the necessary settings and clicks the “Provision Workspace” button, CML will immediately conduct a Validation Check of the provided inputs. Any issues found will be highlighted on the “Provision Workspace” screen, enabling Admins to correct any issues before attempting to provision the ML Workspace again.

Applied ML Prototypes (AMPs) in Cloudera Cloudera AI (CML) enable Users to immediately get started with powerful end-to-end examples of Projects that include all the dependencies, algorithms, and user applications required to directly apply use cases to their own needs. While CML ships with access to a library of AMPs created and maintained by Cloudera’s own Fast Forward Labs team, Admins can now set up customized AMPs that are stored in Azure Repos. When defining the AMP Catalog Sources, Admins have been able to define sources through a Git Repository URL or Catalog File URL, pointing at repositories internal to the organization. Admins can now point these sources to repositories stored in Azure Repos, in addition to other repositories such as GitHub.To learn more, visit the documentation about AMP Catalogs.

ML Runtimes are designed to be the successor to Engines in Cloudera Cloudera AI (CML), moving away from the monolithic architecture of Engines and enabling finer grained control over exactly what is necessary by defining the Editor, Kernel, Edition, and Version needed for the given workload. One core piece previously missing from ML Runtimes was the ability to define Addons that go beyond these settings, in particular the ability to leverage the Spark execution engine through an ML Runtime. Admins can now define Addons for ML Runtimes, including Spark and Hadoop CLI. In doing so, Users can leverage these Addons when running workloads such as Sessions, thereby enabling such functionality as Spark-on-K8s to be run through the ML Runtime.To learn more, visit the documentation about Runtime Addons in CML.

Previously, Admins could assign roles such as “MLUser” at the Environment level, enabling the Users to access the ML Workspaces under that Environment. However, those users would not show up inside the ML Workspace until they log in for the first time, preventing other Users from taking actions such as proactively adding them to Projects. From the Users page, Admins can now sync the list of Users between the CDP Management Console and the given ML Workspace. After doing so, any users who have been assigned access to the ML Workspace through the “MLAdmin”, “MLUser”, or “MLBusinessUser” role would appear appropriately within the ML Workspace. This enables other Users to manage any new Users who have not logged in yet as needed, such as adding them to necessary Teams, Projects, etc.To learn more, visit the documentation about Synchronizing Users in CML.