What's New @ Cloudera

The Data In Motion Team is pleased to announce the release of Cloudera Data Flow 3.2 for Cloudera on cloud.In this release, we’ve focused on major architectural modernization and eliminating friction across the development lifecycle. By introducing zero-touch, cross-cluster authentication, smarter cloud resource management, and a massive upgrade to our underlying infrastructure, we are delivering a more resilient, secure, and intuitive Cloudera Data Flow.Release Highlights:Frictionless Cross-Cluster Communication: We introduced DefaultNodeSSLContextService. This allows you to seamlessly send data from one NiFi cluster (deployment or test session) to another within the same Cloudera Data Flow environment, completely removing the overhead of manually managing and rotating certificates.Optimized Parameter Groups in Drafts: Drafts that import Shared Parameter Groups benefit from dramatically improved performance, timeout resilience, and reliability when managing parameters at scale.Primary

The Data In Motion Team is pleased to announce the release of Cloudera Flow Management Operator for Kubernetes version 3.3. This release focuses on improving security, streamlining deployments, and enhancing upgrade paths. The core highlights include first-class Apache Ranger authorization, seamless inter-cluster communication, declarative flow deployments, and rolling upgrades for NiFi 2 clusters.Release Highlights:Unified Security Governance Across NiFi Clusters: Centralized, policy-based access control for Operator-Managed NiFi clusters is now possible by integrating customer-managed Apache Ranger with the Cloudera Flow Management Operator for Kubernetes. Automated, Secure Cross-Cluster Communication: The Operator now automatically provisions SSL context services using existing node keystores and truststores, eliminating manual TLS configuration for secure cluster-to-cluster data movement.Zero-Downtime Rolling Upgrades for NiFi 2: Upgrade between NiFi 2 versions smoothly withou

Cloudera’s Data In Motion Team is pleased to announce the release of the Cloudera Streams Messaging Operator for Kubernetes v1.7. This milestone release centers on a major platform update to Strimzi 1.0.1 and Apache Kafka 4.2.0, mandatory API standardization, high-availability maintenance tooling with Strimzi Drain Cleaner, refined Kafka configuration controls, and a redesigned Cloudera Surveyor experience.Release HighlightsRebase to Strimzi 1.0.1 and Kafka 4.2.0This release updates the Operator to Strimzi 1.0.1 (up from 0.49.1) and Apache Kafka 4.2.0 (up from 4.1.1). This upgrade delivers upstream enhancements, including per-listener Kafka configurations, advertised port templates for predictable external broker port mapping, environment variable-based rack awareness without requiring ClusterRoleBindings, and on-demand KafkaUser certificate renewal via annotations.Strimzi v1 API EnforcementThe v1 API  is now the sole supported API version for all Strimzi Custom Resource Definitio

Bridging the Visibility Gap for Legacy and Niche SystemsWhile Cloudera Data Lineage  provides native harvesting for various BI systems, many organizations rely on niche, homegrown, or legacy tools that aren't covered by our standard support. To bridge this gap, we previously offered the "Universal Connector," enabling users to import lineage from ETL tools, reports, or databases using CSV templates.However, a significant hurdle remained: while SQL scripts power many processes and reports, users were forced to manually define every source column and its associated logic. Furthermore, the platform lacked a way to represent the actual scripts or the specific logic they contained.The Solution: Custom Lineage ConnectorThe latest update to this capability introduces the ability to include Source SQL scripts directly within CSV files. This update allows direct injections of source SQL queries for databases, ETL/ELT tools, and BI tools via CSV. To emphasize this new flexibility and acknow

The Data In Motion Team is pleased to announce the release of Cloudera Flow Management Operator for Kubernetes version 3.1. This release introduces powerful new security and automation features, focusing on programmatic identity and access management, automated certificate generation, and intelligent cluster lifecycle enhancements that significantly reduce admin overhead and streamline user onboarding.Release Highlights:Programmatic User Group Management via UserGroup Custom Resource Definition (CRD): This new CRD enables admins to configure NiFi user groups directly in YAML. By managing user groups as code within the Kubernetes environment, it streamlines the organization of users, reduces manual configuration, and ensures consistent group management across clusters, significantly improving overall admin efficiency.Streamlined Access Control with AccessPolicyProfile CRD: This new CRD allows admins to define sets of access policies and roles for users programmatically. By consolidating

The Data In Motion Team is pleased to announce the release of Cloudera Data Flow 3.1 for Cloudera on cloud.In this release, we’ve focused on removing some friction that slows teams down. By introducing command line interface (CLI) automations and advanced debugging tools, we’re giving developers more visibility and admins more control. These efficiency gains are backed by a leaner, more performant Flow Designer and a hardened foundation that stays ahead of evolving security standards.Release Highlights:Command-Line Test Session Management: Users can now manage Flow Designer test sessions via the CLI. This update enables seamless automation of test session  workflows, eliminating repetitive manual steps in the UI.Streamlined Parameter Orchestration: We’ve extended full CLI support to Shared Parameter Groups, including the ability to reference them by CRN. By removing the need for exhaustive group details during CLI deployment and enabling full Create, Read, Update, Delete (CRUD) op

This update allows organizations to move beyond reactive troubleshooting by providing a live, continuous pulse of their data ecosystem. By delivering instant visibility into infrastructure, services, and active workloads, RTM ensures that your platform remains healthy, performant, and reliable.Eliminate Blind Spots with Proactive VisibilityIn high-scale data environments, even a few minutes of downtime or a runaway query can derail critical business operations. RTM addresses this by monitoring three core areas:Cluster Infrastructure: Track the health of your underlying hardware and cloud resources (e.g., CPU, memory, and disk I/O) to spot resource exhaustion before it triggers a crash.Service Health: Monitor the uptime and performance of vital services like Hive, Impala, and Spark in real time.Workload Performance: Identify failing jobs or deviating queries as they happen, allowing for immediate intervention to prevent cascading failures across the cluster.Key Benefits:Proactive Mitiga

Cloudera’s Data In Motion team is pleased to announce the release of Cloudera Streaming Analytics 1.17.0 for on-premises deployments. This release focuses on enterprise-grade stability, enhanced security protocols for Kafka integration, and a modernized approach to extensibility through Python.Release HighlightsJava Upgrade to JDK 17CSA 1.17.0 is now built and verified using JDK 17, providing better performance and long-term support for modern Java applications.Proactive Vulnerability ManagementThis release includes extensive core dependency upgrades specifically targeted at resolving CVEs, ensuring your production streaming pipelines remain secure and compliant.Enhanced Kafka Data Source ConfigurationsStreaming data pipelines require precision. Cloudera Streaming Analytics 1.17.0 now allows users to specify custom properties for Kafka connections directly within the Kafka Data Source form. Furthermore, we have introduced explicit support for JKS, PKCS12, and BCFKS truststore and keyst

Native Governance and Data Lineage for the Unified Data FabricThe Cloudera Data Lineage team is happy to announce that Cloudera Data Lineage now supports Trino federated query engine for big data.This integration allows Trino users to access data wherever it lives without sacrificing governance, trust, or visibility.As modern enterprises navigate increasingly complex, hybrid, and multi-cloud environments, data often becomes fragmented across various systems and silos. Combining the federated query engine of Trino with the automated lineage of Cloudera Data Lineage provides organizations with unprecedented insight and  visibility.Here are the primary benefits and use cases that the  integration of Cloudera Data Lineage with Trino provides:Interoperability and Federated Access: Cloudera Data Lineage's newly released Trino integration enables customers to federate queries and securely access data from both Cloudera and non-Cloudera engines. This capability allows organizations t

Cloudera’s Data In Motion Team is pleased to announce the release of the Cloudera Streaming Analytics Operator for Kubernetes v1.5. This release advances our cloud-native streaming capabilities by introducing the Materialized View engine to Kubernetes, optimizing job handling responsiveness, and updating core infrastructure components for better performance and security.Release HighlightsMaterialized View Engine on Kubernetes Cloudera SQL Stream Builder now supports the Materialized View (MV) engine within Kubernetes deployments. This allows users to create and maintain mutating snapshots of queried data, which can be seamlessly accessed via REST APIs, bridging the gap between real-time streams and request-response applications.Asynchronous Flink Job Handling To improve the user experience, SQL Stream Builder now handles Flink jobs asynchronously. This architectural change significantly enhances the responsiveness of the Streaming SQL Console and ensures more stable job statu

We are thrilled to announce the general availability of Cloudera Data Services On Premises 1.5.5 Service Pack 2 (SP2)!This release is packed with massive updates designed to accelerate your AI journey, boost your analytics performance, and lock down your on-premises data environments. Whether you want to deploy LLMs securely or speed up your SQL queries, 1.5.5 SP2 brings the capabilities you need to modernize your data ecosystem.💥 Release HighlightsDeploy Private GenAI at Scale: Powered by NVIDIA, Cloudera AI Inference is now Generally Available (GA). You can now securely deploy and scale private GenAI applications directly behind your firewall, maintaining strict 100% data sovereignty.Blazing-Fast SQL with Trino: Trino is now fully GA within Cloudera Data Warehouse! Take advantage of highly efficient, federated distributed SQL querying across your entire data estate.Enhanced Observability: Say goodbye to operational blind spots. Faster troubleshooting is now possible via full, n

Release HighlightsRebase to Strimzi 0.49.1 and Kafka 4.1.1This release of the Operator is now based on Strimzi 0.49.1 (up from 0.47.0) and Apache Kafka 4.1.1 (up from 4.0.1). This rebase brings the latest performance optimizations and feature sets from the upstream community, ensuring your streaming infrastructure remains at the cutting edge.Introduction of Cloudera Schema RegistryNew in version 1.6, Schema Registry is now available as a standalone application that allows you to efficiently store and manage schemas for your streaming data. It supports Avro and JSON schema formats, and provides schema evolution capabilities with configurable compatibility modes. Schema Registry helps ensure data consistency across your streaming applications by providing a central repository of schemas that applications use to validate their data. It provides a REST API and client libraries for programmatic access.Apache Iceberg Sink Connector CertificationTo empower modern data lakehouse architectures,

Our goal is to provide a seamless, high-performance environment that bridges the gap between raw data and actionable insights. By integrating AI-driven SQL conversion and strengthening our connector suite, we are empowering teams to work faster, more securely, and with greater visibility across the entire data lifecycle.To better align with Cloudera’s suite of solutions, the Octopai convention and reference will now be labelled Cloudera Data Lineage. All materials referencing Cloudera Data Lineage relate to what was formerly Octopai.New FeaturesThis release focuses on three core pillars: advanced cloud-data integration, enterprise-grade security, and robust pipeline connectivity.Enterprise-Grade AuthenticationThe following releases prioritize support for Cloudera authentication protocols.Spark Lineage + Kerberos Authentication Support – Secure your Spark lineage leveraging secure integration with industry-standard Kerberos authentication.Secure Hive & Impala Kerberos Integration –

Cloudera’s Data In Motion Team is pleased to announce the release of the Cloudera Streaming Analytics - Kubernetes Operator 1.4. This release focuses on modernization through Flink and Flink Operator rebases, enhancing security with FIPS compliance, and improving developer productivity by enabling Python UDFs by default.Release HighlightsRebase to Flink Operator 1.11 and Apache Flink 1.20This release of the Operator is now based on the Apache Flink Kubernetes Operator 1.11 and Apache Flink 1.20. These rebases bring the latest features, performance optimizations, and stability fixes from the upstream communities, ensuring your streaming workloads benefit from the newest advancements in high-performance stream processing.FIPS Mode Support To meet the strictest security and compliance requirements, the Cloudera Streaming Analytics - Kubernetes Operator can now be configured and deployed in Federal Information Processing Standards (FIPS) mode. This ensures that your deployment utilize

Cloudera’s Data In Motion Team is pleased to announce the release of the Cloudera Streams Messaging - Kubernetes Operator 1.5. This release centers on modernization, adopting the latest Kafka standards, enhancing security, and significantly improving the usability of the Cloudera Surveyor management tool.Release HighlightsRebase to Strimzi 0.47.0 and Kafka 4.0.1This release of the Operator is now based on Strimzi 0.47.0 (up from 0.45.0) and Apache Kafka 4.0.1 (up from 3.9.0). These rebases bring the latest features, performance optimizations, and stability fixes from the upstream communities.FIPS 140-2/3 Compatibility and ComplianceTo meet the strictest security requirements, Cloudera Streams Messaging - Kubernetes Operator components are now FIPS 140-2/3 compliant. The core Operator and its managed components will automatically enable FIPS mode on FIPS-enabled hosts.Cloudera Surveyor is also FIPS compliant, though it requires you to manually enable FIPS mode through configuration. Sur

We are excited to announce the release of Cloudera Distribution of Apache Spark (CDS) 3.5 for Cloudera on premises 7.1.9. This new release is available as an add-on parcel and provides long-term support for Apache Spark 3.5, the latest Long-Term Support (LTS) version adopted by both Cloudera and the broader Apache community.What's Changing?This update is for customers using the CDS 3.3 add-on parcel to use Spark 3 with Cloudera on premises 7.1.9.  Key Benefits of Upgrading to Apache Spark 3.5Cloudera recommends upgrading to CDS 3.5 for all users currently running CDS 3.3. This upgrade offers significant advantages:Performance Boost: Spark 3.5 is expected to deliver a performance increase of over 20% compared to Spark 3.3. This means faster data processing and more efficient workloads for your enterprise.Enhanced Features: This release includes enhanced security, bug fixes, and support for newer versions of Python. These improvements ensure your data engineering workflows are

Cloudera’s Data In Motion Team is pleased to announce the release of the Cloudera Streaming Analytics - Kubernetes Operator 1.3, an integral component of Cloudera Streaming - Kubernetes Operators. This release includes rebases to Apache Flink 1.20 and Apache Flink Kubernetes Operator 1.11.0. Other changes and updates are focused on enhancing security, usability, and making the product more robust.Release HighlightsRebase to Flink 1.20For more information, see the Flink 1.20 Release Announcement and the Release Notes.Rebase to Flink Kubernetes Operator 1.11.0For more information, see the Apache Flink Kubernetes Operator 1.11.0 Release Announcement and the Release Notes.Flink OpenTelemetry Metrics Reporter (Technical Preview)The OpenTelemetry metrics reporter is now included, in Tech Preview, in the operator image. It makes it easier and more efficient to aggregate job metrics to a central service like Prometheus or any OpenTelemetry-compatible service, and aggregate metrics using open s

The Data In Motion team is thrilled to announce the release of Cloudera Streams Messaging - Kubernetes Operator 1.4 and the exciting new Cloudera Surveyor! This release marks a significant step forward in simplifying and enhancing your Apache Kafka management experience on Kubernetes, focusing on intuitive user experience and powerful new capabilities.Release HighlightsIntroducing Cloudera Surveyor: Your New Go-To for Seamless Kafka Management on Kubernetes Cloudera Surveyor is a brand-new, intuitive graphical interface built to streamline the monitoring and management of your Apache Kafka clusters running on Kubernetes. As a core component delivered with Cloudera Streams Messaging - Kubernetes Operator 1.4, Surveyor empowers you with:Effortless Management: Move beyond the command line with a modern, easy-to-use UI for all your Kafka operations.Multi-Cluster Visibility: Manage and monitor multiple Kafka clusters from a single, centralized console.Actionable Insights: Get real-time

Cloudera Data Engineering 1.24.1We are excited to announce the latest release of Cloudera Data Engineering on Cloud. This release introduces advancements focused on optimizing cost, enhancing security, and improving the overall developer and administrative experience.Key FeaturesSuspend and Resume (Technical Preview): Temporarily suspend your entire Cloudera Data Engineering Service during idle periods to achieve up to 80%  cost savings.Git Integration Support (GA): The Git integration feature is now generally available.  Stay tuned for more enhancements in the upcoming releases.Security Hardened Images:  As part of our ongoing effort to improve security and resilience against Common Vulnerabilities and Exposures (CVEs), new hardened images are used by default during service and virtual cluster creation.  In addition, new hardened images for Spark and Airflow will be available that contain runtime components upgrades, including Python and JDK. For more information,

We are excited to announce long-term support for Apache Spark 3.5 in the Cloudera Unified Runtime (7.3.1 SP1). This update brings the newest Spark version and a more modern, high-performance experience for your data engineering teams. Key BenefitsUp to 50% faster long-running queries, with 5-10% overall performance gains from advanced query planning, adaptive execution, and dynamic partition pruning.Modern developer experience with wider Python support (PySpark) and broader SQL coverage.Stronger stability and scalability with improved memory management, fault tolerance, and support for large-scale workloads.Next StepsAs of Cloudera Runtime 7.3.1.200 SP1, Spark 3 is the default version, and Spark 2 has been fully removed.If you already use Spark 3, upgrading is simple – just recompile your applications for Spark 3.5 and continue running your jobs. Migrating Spark Applications has useful resources to consider when moving between versions.Migrating from Spark 2? Start with these reso

SELinux allows you to set access control through policies. You can set the SELinux mode while creating a new operational database. You can configure the SELinux enforcement for your Cloudera Operational Database using the Cloudera Operational Database UI or CDP CLI.This feature is under technical preview. To use this feature, you must have the CDP_SECURITY_ENFORCING_SELINUX entitlement in your Cloudera environment and the minimum runtime version of 7.2.18.700 or 7.3.1.100. Contact Cloudera support if you do not have this entitlement.Using the CDP CLI:You can define the SELinux mode using the seLinux parameter in the create-database command. The supported SELinux modes are:ENFORCING: Enables SELinux in enforced mode, actively applying security policies.PERMISSIVE (default): Sets SELinux to permissive mode, logging any security violations without enforcing policies.If you do not define the seLinux parameter, by default, the PERMISSIVE mode is applied.The following example shows usage of

Cloudera Operational Database cluster deployments on AWS Graviton environments are now generally available. AWS Graviton is a family of general-purpose, ARM-based processors designed for cloud workloads. Cloudera Operational Database supports the following Graviton versions: Graviton2: Used in Cloudera Operational Database deployments on I4G instances Graviton4: Used in Cloudera Operational Database deployments on I8G instances AWS Graviton processors deliver exceptional price performance for workloads running on AWS EC2. With Graviton4, you can further optimize costs while achieving superior performance. For additional details, see the AWS press release. To help you choose the most suitable ARM processor for your performance requirements, consult the AWS Graviton Processors documentation, which provides a detailed comparison between the Graviton2 and Graviton4 instances. For more information on AWS Graviton support, see AWS Graviton instances in Cloudera Operational Database. This f

The Data In Motion team is pleased to announce the release of Cloudera Streaming Analytics 1.15 for Cloudera  7.3.1. This release  focuses on enhancing the user experience and adding new important features to the product, and includes improvements to SQL Stream Builder as well as a rebase to Apache Flink 1.20.1..Release HighlightsCloudera platform supportCloudera Streaming Analytics 1.15 is supported on Cloudera 7.3.1.100 (Cumulative Hot Fix 1). Ensure that you review the 7.3.1.100 Release Notes and Support Matrix to understand which operating system, database, and JDK versions are supported for Cloudera Streaming Analytics as well.Rebase to Apache Flink 1.20Streaming analytics deployments, including SQL Stream Builder, now support Apache Flink 1.20. For more information on what is included in the Apache Flink 1.20 version, see the Apache Flink 1.20 Release Announcement and Release Notes.Support for batch-mode queries in Cloudera SQL Stream BuilderUsers can select and use “ba

Release HighlightsRebase on Flink 1.19.2For more information, see the Flink 1.19.2 Release Announcement and the Release Notes.General availability for Cloudera SQL Stream BuilderCloudera SQL Stream Builder, previously in Technical Preview, is now generally available in Cloudera Streaming Analytics - Kubernetes Operator.Cloudera SQL Stream Builder is a comprehensive interactive user interface for creating stateful stream processing jobs using SQL. For more information about SQL Stream Builder and its features, see the Getting started with SQL Stream Builder page.LDAP authentication in SQL Stream BuilderLDAP-based authentication is available for Cloudera SQL Stream Builder. For more information, refer to LDAP authentication.Custom truststoresYou can now specify custom truststores when installing the Cloudera Streaming Analytics - Kubernetes Operator. For more information, refer to Security configurations.Secure TLS connections to the SSB UI and APIUsers can connect to Cloudera SQL Stream

Cloudera’s Data In Motion Team is pleased to announce the release of Cloudera Streams Messaging - Kubernetes Operator 1.3, an integral component of Cloudera Streaming - Kubernetes Operator. With this release, customers receive a rebase to Kafka 3.9, automatic cluster rebalance, better offset management capabilities for Kafka connectors, and more!Release HighlightsRebase to Kafka 3.9For more information, see the Kafka 3.9 Release Notes and the list of notable changes.Rebase to Strimzi 0.45.0For more information, see the Strimzi 0.44.0 Release Notes and Strimzi 0.45.0 Release Notes.KRaft (Kafka Raft) is generally availableYou can now deploy Kafka clusters that use KRaft instead of ZooKeeper for metadata management. Additionally, you can migrate existing ZooKeeper-based Kafka clusters to use KRaft. With the addition of KRaft, ZooKeeper is deprecated. Deploying new or using existing Kafka clusters running in ZooKeeper mode is deprecated. Additionally, ZooKeeper will be removed in a future