What's New @ Cloudera

 This release (2.8.0-b274) of Cloudera DataFlow (CDF) on Cloudera on cloud introduces the ability to change the flow definition version of running deployments, a new Overview page with tutorials and shortcuts, the ability to create NiFi 2.0 deployments (Technical Preview), filtering ReadyFlows by different categories, new GenAI ReadyFlows, and supports new Kubernetes versions.Key Deployment & Platform features for this releaseLatest NiFi versions:Flow Deployments now support NiFi 1.25 and NiFi 2.0Change Flow Version of existing deployments:You can now change the flow definition version of existing deployments. Changing flow versions preserves state and removes the need for tedious redeployments. Filtering & Tagging in ReadyFlow Gallery: The ReadyFlow gallery has been improved and customers can now filter for different categories, sources, destinations and compatible NiFi versions.New ‘Overview’ page: When navigating to CDF, users land in the new ‘Overview’ page which

In CDP CLI, while creating an operational database, you can specify the encryption key to encrypt the volume for instance groups using the --volume-encryptions (array) option. You can select this option only in AWS environments.Following is a sample command.opdb create-database --environment-name <environment-name> --database-name <database-name> --disable-external-db --scale-type MICRO --attached-storage-for-workers '{"volumeCount":1,"volumeType":"SSD","volumeSize":100}' --endpoint-url http://localhost:8988 \--volume-encryptions '[{"encryptionKey": "<aws-key-arn>","instanceGroup": "GATEWAY"}]'Shorthand syntax:encryptionKey=string,instanceGroup=string ... (separate items with spaces)JSON syntax:[{"encryptionKey": "string","instanceGroup": "WORKER"|"LEADER"|"MASTER"|"GATEWAY"|"STRONGMETA"|"EDGE"}...]For more information, see CDP CLI documentation.This is available starting in COD v1.41 and you do not need to upgrade to a higher version of COD to benefit from this featu

When you create and deploy an operational database in an Azure environment, by default, COD clusters now use Standard_D8s_v3 instance type instead of Standard_D8_v3. The instance type is upgraded to support encryption at the host level.If you want to retain the Standard_D8_v3 instance type, you must have the COD_USE_DV3_INSTANCE_TYPE entitlement on your account.This is available starting in COD v1.41 and you do not need to upgrade to a higher version of COD to benefit from this feature.Try the Cloudera Operational Database for free with Test Drive (no cloud account or setup is needed).

In CDP CLI, while creating an operational database, you can set the default root volume size with the --root-volume-size (integer) option in GiB for all the instances in the cluster.Following is a sample command.cdp opdb create-database --environment-name test-env --database-name test-db --root-volume-size 300For more information, see CDP CLI documentation.This is available starting in COD v1.41 and you do not need to upgrade to a higher version of COD to benefit from this feature.Try the Cloudera Operational Database for free with Test Drive (no cloud account or setup is needed).

When you create an operational database with HDFS storage type, the COD clusters now use 16 core instances on AWS, Azure, and GCP environments for worker nodes. The COD clusters with HDFS storage type are upgraded to enhance the performance and usability of the COD.The new worker instances for HDFS storage type are as follows:AWS: m5.4xlargeAzure: Standard_D16_v3GCP: n2-standard-16This is available starting in COD v1.41 and you do not need to upgrade to a higher version of COD to benefit from this feature.Try the Cloudera Operational Database for free with Test Drive (no cloud account or setup is needed).

We're pleased to announce the release of Cloudera Data Platform (CDP) Private Cloud Base 7.1.7 Service Pack 3 (SP3), bolstering this Long Term Supported (LTS) version of the platform further. This service pack strengthens the platform's reliability and security posture.Highlights of SP3 include:Enhanced data security and InfoSec compliance with 100+ resolved critical Customer Vulnerabilities and Exposures (CVEs) and fosters customer confidence while safeguarding their data assets. CVEs have been resolved in some of the prominent frameworks including: Netty/Jetty, Jackson databind, Guava, Apache Tomcat and Apache Shiro, Snappy Java, and Spring framework.Enhanced business continuity and data integrity by prioritizing stability and reliability to better protect the operation and integrity of your critical data, as well as other improvements such as new properties enhance retry mechanism for Kafka / Kafka Connect Ranger plugins for better handling of comm

We’re excited to share the latest runtime version of Cloudera on public cloud: 7.2.18! This update is now GA, and it brings enhanced security, efficiency, and data management capabilities to your cloud environment, ensuring your infrastructure is fully supported and optimized for future innovation.What’s New?RHEL 8 is the new default OS, bringing a secure, high-performance, and cloud-optimized foundation for your big data workloadsRolling Upgrades, now GA for Data Lakes and Cloudera Operational Database, allow OS and runtime upgrades without any application downtimeApache Iceberg + Apache Atlas Integration is now GA for metadata and lineage support for governance of Iceberg tablesNew Flow Management Data Hub Tech Preview templates, based on Apache NiFi 2.0, enable you to build GenAI pipelines easier and faster using the latest version of Apache NiFiCloudera Streaming Analytics 1.12 is now available on the Streaming Analytics DataHub, with Apache Flink 1.18, Flink Application Mode, PyFl

When you create an operational database using the COD UI, the new Create Database option provides enhancements to the database creation workflow. In addition to the enhanced look and feel for better usability, the workflow also provides two additional steps called, Settings and Review.In the Settings step, you can view the default settings for your database and can also modify them. In the Review step, you can view, and finalize your changes, and also obtain a complete CLI command, in case you want to create the database using CDP CLI. Documentation link: Creating a database using CODThis is available starting in COD v1.40 and you do not need to upgrade to a higher version of COD to benefit from this feature.Try the Cloudera Operational Database for free with Test Drive (no cloud account or setup is needed).

Hello CDH Users,We are pleased to announce the release of Cloudera JDBC Connector for Apache Hive 2.6.25. This release has the following fixes and enhancements:Released 2024-02-05Resolved IssuesThe following issue has been resolved in Cloudera JDBC Connector for Apache Hive 2.6.25.  * [HIVEJ-790] When connecting to a CDW cluster using HTTP transport, the    connector returns an error.Workflow Changes ============================================================= The following changes may disrupt established workflows for the connector. 2.6.23 -----------------------------------------------------------------------  * [HIVEJ-761] Removed support for Java 7    Beginning with this release, the connector no longer supports Java 7.For a    list of supported JDBC versions, see the Installation and Configuration    Guide.   * [HIVEJ-649] Removed support for Hive Serv

We are thrilled to announce the latest release of Cloudera Data Engineering for the Public Cloud. This release marks a significant milestone towards achieving parity with Spark-on-Base & Datahub. The highlight of this release is GA of In-place Upgrade, with auto backup, providing a streamlined experience for customers to transition to the latest version of CDE. Interactive Session, featuring both CLI and UI-based development, is also now GA. We are officially introducing the 2-tier pricing - CDE Core and CDE All-purpose. CDE Core virtual clusters represent our current functionality that is designed for long-running, operational pipelines. CDE All-purpose cluster includes a broader set of features compared to CDE Core including Interactive Sessions. This gives customers greater flexibility to manage costs by tailoring workloads to specific compute tiers.  Key features for this releaseHomepage 2.0 Complete revamp of the land

We are thrilled to announce the latest release of Cloudera Data Engineering for the Public Cloud. This release marks a significant milestone towards achieving parity with Spark-on-Base & Datahub. The highlight of this release is GA of In-place Upgrade, with auto backup, providing a streamlined experience for customers to transition to the latest version of CDE. Interactive Session, featuring both CLI and UI-based development, is also now GA. We are officially introducing the 2-tier pricing - CDE Core and CDE All-purpose. CDE Core virtual clusters represent our current functionality that is designed for long-running, operational pipelines. CDE All-purpose cluster includes a broader set of features compared to CDE Core including Interactive Sessions. This gives customers greater flexibility to manage costs by tailoring workloads to specific compute tiers. Key features for this releaseHomepage 2.0 Complete revamp of the landing page with a

Hello CDH Users, We are pleased to announce the release of the New Cloudera ODBC Connector 2.7.1 for Impala. This release has the following fixes and enhancements: Enhancements & New Features  [IMP-965] Multi-socket fetch support:   You can now retrieve multiple row sets from the server concurrently using multiple client connections. To do this, set EnableMultiSocketFetch to 1 [IMP-963] Improved fetch performance: The connector now has enhanced fetch performance [IMP-976] Updated third-party library: The connector now uses libcURL 8.4.0. Previously, the connector used version 8.1.2 Resolved Issues The following issue has been resolved in Cloudera ODBC Connector for Apache [IMP-974] When authenticating with SAML and an Impala CDW cluster is configured to Active-Active HA mode, the connector returns an error  Known Issues The following are known issues that you may encounter due to limitations in the data source, the connector, or an a

The latest release (2.7.0-b190) of Cloudera DataFlow (CDF) on Cloudera on cloud introduces new Change Data Capture processors, flow version tagging, deployment configuration reuse, NiFi bulletin monitoring and supports new Kubernetes versions. Key Deployment & Platform features for this releaseLatest NiFi versionFlow Deployments and Test Sessions now support the latest Apache NiFi 1.24 release.New Debezium CDC processorsCustomers can now build and deploy flows with Debezium based CDC processors for MySQL, Postgres, Oracle, SQLServer and DB2 databases.Deployment alerts for NiFi bulletinsCDF now detects bulletin error messages in NiFi flow deployments and displays them as a warning in the dashboard.Flow Definition versions can now be taggedCustomers can now use tags to identify versions easier. Common use cases for tags are applying a custom versioning scheme, or labeling flow versions as ‘ready for deployment’. The CLI introduces new commands to search flow definitions for tags

When you create a new operational database using CDP CLI, you can enable fast autoscaling by defining the required parameters using the --auto-scaling-parameters option. COD now supports a new instance group called, Compute. The nodes under this instance group are automatically scaled up or scaled down based on the CPU utilization and RPC latency.To use fast autoscaling, you must have the COD_USE_COMPUTE_ONLY_NODES entitlement.Following is a sample command.cdp opdb create-database —environment-name <env_name> –database-name <db_name> --auto-scaling-parameters '{"minComputeNodesForDatabase":<min_compute_nodes>, "maxComputeNodesForDatabase": <max_compute_nodes>}'Documentation link: Fast autoscaling in CODThis is available starting in COD v1.37 and you do not need to upgrade to a higher version of COD to benefit from this feature.Try the Cloudera Operational Database for free with Test Drive (no cloud account or setup is needed).

When you create a new operational database with Cloud With Ephemeral Storage as the storage type on an AWS environment, COD creates the database with an I4i instance type for the worker nodes.This is available starting in COD v1.37 and you do not need to upgrade to a higher version of COD to benefit from this feature.Try the Cloudera Operational Database for free with Test Drive (no cloud account or setup is needed).

COD ensures high availability and fault tolerance using Multi-AZ deployments. A Multi-AZ deployment means that compute infrastructure for HBase’s master and region servers are distributed across multiple AZs ensuring that when a single availability zone has an outage, only a portion of Region Servers is impacted and clients automatically switch over to the remaining servers in the available AZs.Multi-AZ for COD is now supported on Microsoft Azure environments as a technical preview and is considered under development.Documentation link: Multi-AZ deployment on CODThis is available starting in COD v1.37 and you do not need to upgrade to a higher version of COD to benefit from this feature.Try the Cloudera Operational Database for free with Test Drive (no cloud account or setup is needed).

COD has removed the OPDB_USE_EPHEMERAL_STORAGE entitlement because it is not needed anymore. The use of COD on a cloud storage with ephemeral cache is enabled without an entitlement depending on the cluster creation parameters.This is available starting in COD v1.37 and you do not need to upgrade to a higher version of COD to benefit from this feature.Try the Cloudera Operational Database for free with Test Drive (no cloud account or setup is needed).

COD has removed the COD_STOREFILE_TRACKING entitlement because it is not needed anymore. The Store File Tracking (SFT) functionality is enabled on all new COD clusters created with cloud storage.This is available starting in COD v1.37 and you do not need to upgrade to a higher version of COD to benefit from this feature.Try the Cloudera Operational Database for free with Test Drive (no cloud account or setup is needed).

COD has removed the COD_EDGE_NODE entitlement now because it is not needed anymore. COD edge node functionality is enabled for all COD customers now.This is available starting in COD v1.37 and you do not need to upgrade to a higher version of COD to benefit from this feature.Try the Cloudera Operational Database for free with Test Drive (no cloud account or setup is needed).

You can deploy COD on a Google Cloud Platform (GCP) by using Google Cloud Storage (GCS) similar to what is available for Amazon Web Services (AWS) S3 storage and Microsoft Azure blob storage. Now, COD has enabled the COD_ON_GCS entitlement, by default, for such a deployment.This is available starting in COD v1.37 and you do not need to upgrade to a higher version of COD to benefit from this feature.Try the Cloudera Operational Database for free with Test Drive (no cloud account or setup is needed).

We’re excited to announce that Cloudera’s Private Cloud Data Services 1.5.2 is now generally available.  Infuse the power of cloud-native analytics into your on-premises workloads with Cloudera’s industry leading data platform. This release delivers Cloudera’s Open Data Lakehouse on Private Cloud Data Services, providing the scale, speed, and reliability you need to build trusted, enterprise AI to solve your biggest business challenges. Here are some highlights of this release: Apache Iceberg is now generally available, enabling you to implement an Open Data Lakehouse architecture to unify all your data across a hybrid cloud ecosystem.Cloudera Data Catalog is now generally available, making it easier to find the right data at the right time, an essential piece of building out a successful Data Fabric and indeed a Data Mesh.Run new LLM Applied ML Prototypes (AMPs), including parameter efficient fine tuning (PEFT) techniques to greatly decrease the computational and s

The latest release (2.6.1-b92) of Cloudera DataFlow (CDF) on Cloudera on cloud introduces fine-grained access controls for flow drafts and deployments within an environment and support for creating and managing reporting tasks through the CLI.Key Deployment & Platform features for this releaseFine-grained access control for drafts and deployments Create projects, assign drafts and deployments, add CDP users/groups to control access.Reporting Task support Monitor your deployments with NiFi’s out of the box or custom Reporting Tasks! Reporting Tasks can now be added to deployments using new CLI commands cdp dfworkload create-reporting-task Key Flow Designer features for this releaseAdditional Zoom control Users without a scroll wheel or trackpad can now zoom in/out using ‘+/-’ controls on the canvasCollapsible sections in configuration drawer Settings, Scheduling, Properties and Relationships sections can now be collapsed for more efficient flow design To learn more ab

 We are pleased to announce the release of Cloudera JDBC 2.6.23 driver for Apache Hive.This release has the following fixes and enhancements: Enhancements and New Features[HIVEJ-748] Support for JWT authentication[HIVEJ-762] Native GSS-API support[HIVEJ-765] Updated third-party library[HIVEJ-773][02840395] Updated Jackson libraries Workflow ChangesThe following changes may disrupt established workflows for the connector[HIVEJ-761] Removed support for Java 7[HIVEJ-649] Removed support for Hive Server 1.0.0 Getting Started with the Cloudera DriverRead the Cloudera Hive JDBC 2.6.23 Driver for  Apache Hive Release Notes and Installation GuideDownload the connector from the Cloudera Connectors page As always, we welcome your feedback. Please send your comments and suggestions to the user group through our community forums. You can also file bugs through our external Jira projects on issues.cloudera.org.

With the October release of Cloudera Cloudera AI, we have enabled the ability for CDP Machine Users to operate as Service Accounts in an ML Workspace. This feature is available both from within the ML Workspace UI and from CML APIs.The Service Account plays a crucial role in moving ML/AI workloads into production for several reasons. They enable secure access to resources and services without tying these accesses to an individual's credentials. Service Accounts also can take on roles in an ML workspace enabling scenarios where a job, deployed model, or AI application can run under the identity of that Service Account. This enables MLOps and CI/CD workloads to run under the Service Account and not tied to a particular user. Furthermore, they offer traceability, as the actions performed using a service account can be audited and monitored, ensuring accountability and governance.To learn more about this release go to our documentation.

We're excited to announce the General Availability (GA) of the Cloudera Model Registry, your centralized hub for storing, managing, and deploying machine learning models and their associated metadata. This robust tool simplifies the MLOps process, enabling organizations to develop, deploy, and maintain machine learning models effortlessly in a production environment.Cloudera's Model Registry addresses the challenges of fragmentation and lack of visibility in MLOps workflows. Serving as the single source of truth for model versions and lineage, it streamlines workflows, enhancing the traceability and reproducibility of model development.The Model Registry is now officially accessible in CML Public Cloud. To harness the full potential of General Availability (GA), upgrade your CML Workspaces and deploy your new Model Registry!You can find more information about the new Model Registry in our documentation.