Announcements

Cloudera Cloudera AI’s new APIv2 provides all CML users with the ability to programmatically create, read, update and delete projects and workloads, including jobs, models and applications. This means that users can automate creation and setup of projects, or trigger actions such as retraining or deploying a new version of a model as part of the project lifecycle, all from within the product or from an external scheduling or CI/CD tool, using the Python client library or HTTPS REST API. In addition to creating production-ready ML models and applications, machine learning engineers must also take those models and applications to production, which often involves deploying to a different environment. Further, these applications are rarely deployed once and left alone. Rather, many models will be updated with further versions as modellers continue to iterate, or as the profile of incoming data changes. These activities are challenging and susceptible to error if they can only be done

With the cdpctl command line utility, Cloud Administrators can verify compatibility of existing setups with CDP prior to handing over to the CDP Administrator.  cdpctl now adds support for Microsoft Azure in addition to Amazon Web Services.   Cloudera on cloud customers often choose to provision their environments into existing cloud accounts where resources such as networking, IAM and storage are pre-created by IT and vetted for adherence to corporate standards.   Getting this right often takes several iterations between the Cloud Administrator and the CDP Administrator. In larger organizations with well-defined organizational boundaries, this exercise can stretch into days. Cloud Administrators can now download the cdpctl utility and use it to verify compatibility of existing resources with CDP on both Azure and AWS.  The leading cause of CDP provisioning failures are related to network, storage locations or access to these (IAM).  This release

Cloudera Data Engineering (CDE) now supports multi-version Spark pipelines.  Users can easily test and promote Spark 2 workloads to Spark 3 to take advantage of the performance and stability improvement in the latest version of Spark.  (Performance improvement of over 30% based on internal TPC-DS benchmarks) Data engineers can run workloads in both Spark 2 and Spark 3 within the same CDP PC environment, therefore maintaining backwards compatibility with legacy workloads while developing new applications on the latest version of Spark.  Administrators have a new option within the Virtual Cluster creation wizard to choose a Spark version.  Once up and running, users can seamlessly transition to deploying their Spark 3 jobs through the same UI and CLI/API as before, with comprehensive monitoring of their pipelines including real-time logs and Spark UI. To learn more,  visit the documentation.

With the release of CDP 7.2.11, it is now super easy to deploy your custom components on your Flow Management DataHub clusters by dropping your components in a bucket of your cloud provider. Until now, when building custom components in NiFi, you had to SSH to all of your NiFi nodes to deploy your components and make them available to use in your flow definitions. This was adding an operational overhead and was also causing issues when scaling up clusters. From now on, it's easy to configure your NiFi clusters to automatically fetch custom components from an external location in the object store of the cloud provider where NiFi is running. All of your nodes will fetch the components after you dropped them in the configured location. You can find more information in the documentation.

With the release of CDP 7.2.11, you now have the possibility to scale up and down both your light duty and heavy duty Flow Management clusters on all cloud providers. You can find more information in the documentation.

CDP Operational Database expands support for SqlAlchemy APIs with Phoenix with support for the getPrimaryKeys and getIndexInfo from the JDBC API through PhoenixDB (along with the SQL Alchemy wrapper APIs).   getPrimaryKeys API retrieves a description of a table's primary key columns ordered by COLUMN_NAMEgetIndexInfo API retrieves a description of a table's indices ordered by NON_UNIQUE, TYPE, INDEX_NAME, and ORDINAL_POSITION This capability is available in PhoenixDB 1.1.0 and is compatible with all CDP 7.x versions and all CDP Operational Database service versions.

We’re thrilled to announce that all Cloudera on cloud customers now have access to Cloudera DataFlow for the Public Cloud (CDF-PC), the first cloud-native runtime for Apache NiFi data flows.  CDF-PC enables Apache NiFi users to run their existing data flows on a managed, auto-scaling platform. With a streamlined way to deploy NiFi data flows and a central monitoring dashboard, it is now easier than ever before to operate NiFi data flows at scale in the public cloud. CDF-PC provides the following features:Import existing NiFi flows and store them in a central catalogDeploy NiFi flows from the central catalog to auto-scaling Kubernetes clustersAllocate resources and set guardrails for individual NiFi deploymentsDefine and monitor KPIs on key NiFi data flow metricsMonitor all your flow deployments from a central dashboardPush-Button NiFi version upgradesGet started with new data flows by using ReadyFlow templates for the most common data movement patternsConsumption based billin

CDP Operational Database (COD) provides increased visibility into performance with graphs on P99 latency, block cache hit ratio, compaction queue size. P99 get latency is one of the key metrics used to determine auto-scaling behavior in COD.  P99 refers to the maximum latency 99% of the requests incur (i.e., 99% of requests are responded to in less than this latency figure).  In other words, only 1% of the requests are expected to be slower than this metric. Block cache hit ratio refers to the # of requests/minute that are serviced from the block cache instead of object storage.  This is an overall driver of database performance as object storage (S3 on AWS, ABFS on Microsoft Azure) is slow relative to the block cache. Compaction queue size gives an indication of the number of Stores in HBase that have been targeted for compaction.  Try out CDP Operational Database for free with Test Drive (no cloud account or setup needed).

The Impala Virtual Warehouses within Cloudera Data Warehouse (CDW) give users an intelligent way to run fast BI queries. Coordinators are the ones that BI tools, such as HUE or Tableau, connect to. There is either one of these running, or two of them if you choose HA mode. Previously, these nodes were always running so that the BI tool connections would not break, and new queries could always be accepted. Now, CDW is able to terminate Coordinators automatically when all queries stop, and restart them once new queries arrive - all without interfering with BI tool connections. Since Impala Executor nodes already had the ability to auto-suspend, this means that all of the largest (i.e. most expensive) nodes used by the Impala Virtual Warehouse now support auto-suspend, dramatically reducing your CDW cost. This capability is currently released as a Tech Preview feature, so please request the Impala Coordinator Auto-Suspend entitlement from your Cloudera account team if you w

Replication Manager dramatically simplifies setup of CDP Operational Database replication automating most parts of setup including:Deployment of the OpDB Replication plugin which eliminates the need for Kerberos x-realm trustCreation of the /user/hbase directory used for snapshot stagingCreation of the table name on target for replicated tablesConfiguration of HBase to setup peering between source & targetConfiguration of HBase replication to enable replicationCreation of snapshot on source and loading of snapshot on target to ensure all existing data is copied as HBase replication only copies data that has changed after it is enabledAutomated rolling restarts after peering Delivered in COD service version 1.13 (deployed services automatically use the latest release) and Cloudera Runtime 7.2.10.1 (requires DataLake and database upgrade)Try out CDP Operational Database for free with Test Drive (no cloud account or setup needed).

COD can now deploy using HDFS on EBS to provide high-performance storage (single millisecond / low double-digit millisecond latencies for many gets/puts).  Use the CDP CLI Beta to deploy a COD with this capability (e.g., “create-database” with “--use-hdfs” flag) Delivered in COD service version 1.13 (deployed services automatically use the latest release) and requires CDP CLI Beta v 0.9.41Try out CDP Operational Database for free with Test Drive (no cloud account or setup needed).

CDP Operational Database (COD) databases store data in cloud storage. The location of the data is stored in cloud storage is defined by the data access cloud storage location in the CDP environment and the COD database name. When a COD database is dropped, any data stored in the cloud storage is not dropped.  You can delete the data separately if you no longer need it. If a COD database is created using the same name (i.e., against a cloud storage location that already contains COD data from a previously deployed database), COD will automatically re-create the tables with the corresponding data in the new COD instance.  Delivered in COD service version 1.13 (deployed services automatically use the latest release)Try out CDP Operational Database for free with Test Drive (no cloud account or setup needed).

We're pleased to announce the release of Cloudera Private Cloud (PvC) Base 7.1.7, which introduces key new features to improve the management, security, and analytical capabilities of the platform and provides additional platform support. This is a cumulative maintenance release that carries forward the improvements from CDP PvC Base 7.1.6 and prior releases.   In addition to new features and platform support, an important goal of the last few releases has been to make the transition to CDP PvC Base easier by adding more upgrade options, and this release continues to deliver against that goal. Here’s an overview of the new features and improvements we’ve added in this release.   Upgrade enhancementsTo help customers with their move to CDP PvC Base we’ve made the following upgrade enhancements in this release:  In-place upgrades are now available for CDH versions 6.1.x, 6.2.x, and 6.3.x., completing our in-place upgrade vision from all supported C

In Cloudera on cloud, each SDX cluster creates a separate kerberos domain and COD databases deploy within the context of each SDX cluster. Prior to this release, using CopyTable required configuration of kerberos x-realm trust between multiple domains. With this release, the HBase CopyTable can now use the same auth mechanism as the OpDB Replication Plugin to copy data across databases without requiring kerberos x-realm trust. Delivered in Cloudera Runtime 7.2.10.1 (requires DataLake and database upgrade to this runtime version) Try out CDP Operational Database for free with Test Drive (no cloud account or setup needed).

With the cdpctl command line utility, Cloud Administrators can verify compatibility of existing setups with CDP prior to handing over to the CDP Administrator.  This will greatly reduce the probability of errors during onboarding into an existing cloud account.   Cloudera on cloud customers often choose to provision their environments into existing cloud accounts where resources such as networking, IAM and storage are pre-created by IT and vetted for adherence to corporate standards.   Getting this right often takes several iterations between the Cloud Administrator and the CDP Administrator. In larger organizations with well-defined organizational boundaries, this exercise can stretch into days. Cloud Administrators can now download the cdpctl utility and use it to verify compatibility of existing resources with CDP.  This release of cdpctl validates AWS infrastructure against a list of our most common failure scenarios, including verifying networkin

Here are some highlights from the month of June:   Top Activities 240+ new support questions 22 new community articles 880+ new members Top Trending Labels Rank Top Tags No. of Views #1 Hive 213,220 #2 Data Ingestion & Streaming 212,130 #3 Data Processing 199,804 #4 Hadoop Core 126,737 #5 NiFi 113,839 Top Trending Community Articles Rank Community Article Author #1 Spark Memory Management @RangaReddy #2 Accessing Cloudera Data Warehouse Created on Different Cloud Providers using JdbcStorageHandler @carrossoni #3 Qualys Vulnerability Scanner requries exceptions for clusters secured with ZooKeeper ACLs  @rblough #4 Apache Ranger LDAP / LDAPS configuration  @tarekabouzeid91 #5 Monitor a Replication Policy details on a Datalake CDP Cluster  @ShaileshS Our Leaderboards We would like to recognize the below community members and employees for their efforts over the l

Administrators can customize the Cloudera provided ML Runtimes to support Data Scientists’ specific use-cases. They can install additional OS packages, Python and R libraries, third-party drivers to enable connecting to external data stores, or even a new editor to be used. CML now enables registering these custom Runtimes and making them available for Data Scientists to use in their projects.  Data Scientists have specific requirements for their working environments, they require a set of R or Python libraries and ready-made connections to fetch from third-party data stores. With the new feature, administrators can create custom Runtimes that data scientists can use in CML. To learn more, visit the documentation about Customized ML Runtimes.

Cloudera ML Runtimes are the default and recommended solution for running user workloads. New Projects will be created with ML Runtimes configured by default and we recommend migrating existing Projects to use ML Runtimes. Legacy Engines are deprecated and will be removed in a future release but workloads running on them remain fully supported.  To learn more, visit the documentation about Engine deprecation.

Administrators of Cloudera Data Engineering (CDE) on Cloudera on cloud now have access to real-time as well as historical diagnostic log bundles without the need to access S3 or Kubernetes APIs.  Bundles are streamed directly to the users local machine pre-packaged as zip files that can easily be uploaded to Support tickets.   Previously administrators of CDE had to use kubectl to access service pods during troubleshooting scenarios, requiring privileged access and manual steps.   Now, within each CDE service, two types of bundles are available, providing different metrics about the health of the underlying kubernetes cluster and service applications.  The first bundle is a summary snapshot of the state of active pods.  The second, allows the administrator to choose a time period along with specific service applications (such cluster autoscaler) to extract historical logs from. To learn more visit the documentation.

CDP Operational Database Experience (COD) enables customers to use Apache Spark on the same database either directly with Apache HBase (NoSQL) or through Apache Phoenix (SQL) simultaneously through the Spark-Phoenix and Spark-HBase connector enabling customer's whose aggregation pipelines were built using both HBase and/or Phoenix to use the Cloud without code change.  

Cloudera Data Engineering (CDE) now supports multi-version Spark pipelines.  Users can easily test and promote Spark 2 workloads to Spark 3 to take advantage of the performance and stability improvement in the latest version of Spark.  (Performance improvement of over 30% based on internal TPC-DS benchmarks) Data engineers can run workloads in both Spark 2 and Spark 3 within the same CDP PC environment, therefore maintaining backwards compatibility with legacy workloads while developing new applications on the latest version of Spark.  Administrators have a new option within the Virtual Cluster creation wizard to choose a Spark version.  Once up and running, users can seamlessly transition to deploying their Spark 3 jobs through the same UI and CLI/API as before, with comprehensive monitoring of their pipelines including real-time logs and Spark UI. Future release, will include the visual performance profiler into Spark 3 job run details. To learn mor

Users of Cloudera Data Engineering (CDE) on Cloudera on cloud can now deploy, monitor, and schedule data pipelines on Microsoft Azure. Customers can take advantage of a fully managed Spark-on-Kubernetes with autoscaling compute and guard rails to control cost, without the usual platform management overhead.   As opposed to traditional job submissions mechanisms that require direct access to the cluster via edge nodes, data engineers can deploy data pipelines to autoscaling Virtual Clusters through simple, browser-based UI wizards or a full-fledged CLI and API.  Once deployed, CDE optimizes execution through performance metric profiling and real-time monitoring, and provides users with a comprehensive view of their pipelines.  And for further operationalization, a managed Apache Airflow service can be used to orchestrate complex pipelines on a schedule or based on event triggers. Future releases, will add support for Spot and SSD instances, as well as Private Li