We are excited to announce the General Availability of Cloudera Flow Management - Kubernetes Operator and Cloudera Streaming - Kubernetes Operator, allowing Cloudera customers to run Apache NiFi and Apache Kafka on Kubernetes. While Kubernetes started out as a container orchestration platform where customers would mostly run stateless applications and APIs, it has now matured to a point where it can run stateful, large scale data processing frameworks such as Apache NiFi and Apache Kafka. We have seen this shift in Kubernetes usage in our customer base and are excited to meet them where they are. Use Case The Kubernetes Operators allow NiFi and Kafka customers to deploy their workloads on existing shared Kubernetes clusters. Kubernetes Operators automate the installation and lifecycle management of applications on Kubernetes, which significantly lowers cost of operations and allows customers to focus on use case implementation instead of cluster operations. For customers, this means that they can take advantage of: Greater hybrid portability and accelerated deployments via streamlined life cycle management Increased resource efficiency via shared clusters Improved scalability via Kubernetes orchestration tools like OpenShift Release Highlights Flow Management Operator GA (NiFi):  Deploys and manages NiFi and NiFi Registry Supports NiFi 1.25 and offers NiFi 2.0 (Tech Preview) Automatically creates zookeeper-less NiFi deployments Provides cfmctl command line tool for easy installation & setup Manual and (CPU based) auto-scaling of NiFi clusters Increased durability and availability through auto-recovering NiFi pods Streaming Operator GA (Kafka) Deploys and manages Kafka, Zookeeper, and Cruise Control Supports Kafka 3.7 Provides rack awareness and HA Supports various authentication options (OAuth, LDAP, mTLS, PLAIN) Supports authorization through Kafka ACLs Prometheus integration for monitoring Learn More Join us for our release webinar showcasing an end-to-end financial services use case running on the new Kubernetes Operators. Read the Operator documentation for detailed information about pre-requisites, installation and configuration ... View more

  This release (2.8.0-b274) of Cloudera DataFlow (CDF) on CDP Public Cloud introduces the ability to change the flow definition version of running deployments, a new Overview page with tutorials and shortcuts, the ability to create NiFi 2.0 deployments (Technical Preview), filtering ReadyFlows by different categories, new GenAI ReadyFlows, and supports new Kubernetes versions. Key Deployment & Platform features for this release Latest NiFi versions: Flow Deployments now support NiFi 1.25 and NiFi 2.0 Change Flow Version of existing deployments: You can now change the flow definition version of existing deployments. Changing flow versions preserves state and removes the need for tedious redeployments.   Filtering & Tagging in ReadyFlow Gallery: The ReadyFlow gallery has been improved and customers can now filter for different categories, sources, destinations and compatible NiFi versions. New ‘Overview’ page: When navigating to CDF, users land in the new ‘Overview’ page which provides documentation, getting started guides/videos and shortcuts for common actions. Customers will now also be notified about new releases being available. New ReadyFlows for - DB2 CDC to Iceberg (Technical Preview) - DB2 CDC to Kudu - MySQL CDC to Iceberg (Technical Preview) - Oracle CDC to Iceberg (Technical Preview) - PostgreSQL to Iceberg (Technical Preview) - Slack to Pinecone (NiFi 2.0) - SQL Server CDC to Iceberg (Technical Preview) Support for new Kubernetes versions AKS 1.28 / EKS 1.28 ... View more

The latest release (2.7.0-b190) of Cloudera DataFlow (CDF) on CDP Public Cloud introduces new Change Data Capture processors, flow version tagging, deployment configuration reuse, NiFi bulletin monitoring and supports new Kubernetes versions.   Key Deployment & Platform features for this release Latest NiFi version Flow Deployments and Test Sessions now support the latest Apache NiFi 1.24 release. New Debezium CDC processors Customers can now build and deploy flows with Debezium based CDC processors for MySQL, Postgres, Oracle, SQLServer and DB2 databases. Deployment alerts for NiFi bulletins CDF now detects bulletin error messages in NiFi flow deployments and displays them as a warning in the dashboard. Flow Definition versions can now be tagged Customers can now use tags to identify versions easier. Common use cases for tags are applying a custom versioning scheme, or labeling flow versions as ‘ready for deployment’. The CLI introduces new commands to search flow definitions for tags and streamlines the CI/CD process. Deployment configurations can now be exported and reused Customers can now export a configuration archive of existing deployments and reuse them when creating new deployments. This speeds up redeploying new versions of the same flow through the Deployment Wizard by filling in parameter values, KPI configurations etc. from the deployment configuration.  Support for new Kubernetes versions AKS 1.27 / EKS 1.27 New ReadyFlows for Debezium CDC to Kudu (Tech Preview) S3/ADLS to Delta Lake HuggingFace dataset to S3/ADLS S3 to Watson.x   To learn more about this release, see the product documentation,  take the interactive Flow Designer product tour and visit the product page! ... View more

The latest release (2.6.1-b92) of Cloudera DataFlow (CDF) on CDP Public Cloud introduces fine-grained access controls for flow drafts and deployments within an environment and support for creating and managing reporting tasks through the CLI. Key Deployment & Platform features for this release Fine-grained access control for drafts and deployments Create projects, assign drafts and deployments, add CDP users/groups to control access. Reporting Task support Monitor your deployments with NiFi’s out of the box or custom Reporting Tasks! Reporting Tasks can now be added to deployments using new CLI commands cdp dfworkload create-reporting-task  Key Flow Designer features for this release Additional Zoom control Users without a scroll wheel or trackpad can now zoom in/out using ‘+/-’ controls on the canvas Collapsible sections in configuration drawer Settings, Scheduling, Properties and Relationships sections can now be collapsed for more efficient flow design   To learn more about this release, see the product documentation, take the interactive Flow Designer product tour and visit the product page! ... View more

The latest release (2.6.0-b311) of Cloudera DataFlow (CDF) on CDP Public Cloud introduces NiFi 1.23, vertical scaling, several cost optimization features, developer productivity improvements and the ability to collect Diagnostic Bundles through the UI. Key Deployment & Platform features for this release Latest NiFi version: Flow Deployments and Test Sessions now support the latest Apache NiFi 1.23 release. Up to 10% savings on compute resources by removing Zookeeper from deployments and test sessions. CDF now uses a NiFi and Kubernetes native clustering protocol. Up to 50% savings on storage cost (AWS only) by moving from GP2 to GP3 volumes and introducing a new, cost efficient ‘standard’ storage profile New ‘Suspend Deployment’ behavior now terminates associated compute resources resulting in Cloudera and infrastructure cost savings. Manual Vertical scaling allows customers to change the deployment sizes easily after a deployment has been created. Allows customers to start small and easily scale to bigger nodes as needed. Select between ‘standard’ and ‘performance’ storage profiles when creating a deployment based on your needs. Create Diagnostic bundles and attach to support cases for faster support case resolution. Allows customers to easily collect relevant logs without sifting through S3 or Kubernetes pods. The Dashboard now displays alerts for failed Kubernetes resources like NiFi pods, statefulsets etc. This makes it easier to see when there are underlying issues on the K8s cluster. A new ‘workspace view’ for centralized resource management allows users to see all resources (deployments, drafts, inbound connections, custom NAR configs) in a central place. Support for new Kubernetes versions AKS 1.26 / EKS 1.25 Key Flow Designer features for this release Support for sensitive dynamic properties The Flow Designer now supports creation of sensitive dynamic properties for several components like InvokeHTTP and ExecuteStreamCommand Download NiFi app log directly from the test session for easier troubleshooting when access to logs is needed during development  Copy bulletin messages using a ‘copy’ icon to share them with colleagues or support To learn more about this release, see the product documentation, take the interactive Flow Designer product tour and visit the product page! ... View more

The latest release (2.5.0-b210) of Cloudera DataFlow (CDF) on CDP Public Cloud introduces NiFi 1.21, flow metrics driven auto-scaling, advanced UIs for JoltTransformJSON and UpdateAttribute processors, bulk actions in the Flow Designer, UDR support on Azure and makes in-place upgrades generally available to all customers.   Flow Deployments and Test Sessions now support the latest Apache NiFi 1.21 release. In-place upgrades are now generally available to all customers. For more information, see Service upgrade. Flow Deployments now support Flow Metrics Scaling in addition to CPU utilization based auto-scaling. For more information, see Auto-scaling flow deployments When selecting Private Cluster during enablement on Azure, CDF no longer provisions a public load balancer and now supports User Defined Routes (UDR). Users can now provide their own values for the Kubernetes Pod CIDR Range and Kubernetes Service CIDR Range during enablement CDF now inherits load balancer configuration including available subnets from its associated CDP environment during the enablement process New Flow Designer Features The Flow Designer now supports multi-selection on the canvas and bulk actions for Start, Stop, Enable, Disable, Move, Change parent group, Copy/Paste, and Delete. The Flow Designer now supports the advanced configuration UI for UpdateAttribute. The Flow Designer now supports the advanced configuration UI for JoltTransformJson.   The Flow Designer now supports Birdseye and Zoom controls.   The Flow Designer now supports Processor Diagnostics with an active Test Session. New ReadyFlows The following new ReadyFlows have been added to the ReadyFlow Gallery: CDW Ingest CDP Kafka to Snowflake Slack to S3 Updated Confluent Cloud to Snowflake using new Snowpipe processors To learn more about this release, see the product documentation, take the interactive Flow Designer product tour and visit the product page! ... View more

The latest release (2.4.0--b317) of Cloudera DataFlow (CDF) on CDP Public Cloud makes the Flow Designer generally available to all CDP Public Cloud customers, introduces a streamlined way to collect log files, supports new NiFi and Kubernetes versions and introduces several usability improvements when creating and managing flow deployments through the UI.   The ability to design flows with the new Flow Designer is now generally available to all CDP Public Cloud customers. The CDP cli now supports commands to generate diagnostic bundles for CDF for troubleshooting Flow Deployments and Test Sessions now support the latest Apache NiFi 1.20 release When creating a deployment through the deployment wizard, users can now filter for sensitive or empty parameters CDF now creates Kubernetes clusters with version 1.23 in AWS EKS and 1.24 in Azure AKS. Upgrading and restarting deployments now triggers a NiFi Health Alert if the NiFi cluster does not return to a healthy state. This can be cleared by performing a Restart Deployment once the underlying issue has been fixed. Restart Deployment now supports force suspending the NiFi Flow in case the running flow itself is impairing the health of the NiFi cluster. Enable DataFlow action now performs validation steps immediately and fails fast if there are issues with the request Some enablement failure messages are improved when the failure is due to common issues surrounding volume provisioning The following ReadyFlows have been added to the ReadyFlow Gallery: HubSpot to S3/ADLS JDBC to JDBC Kafka to Apache Iceberg Shopify to S3/ADLS To learn more about this release, take the interactive Flow Designer product tour , see the product documentation, and visit the product page! ... View more

The latest release (2.3.0-b347) of Cloudera DataFlow (CDF) on CDP Public Cloud introduces the following new features for both, AWS and Azure customers:   Flow Designer [Technical Preview] Developers can now build new data flows from scratch using the integrated Designer. Flow Drafts can be tested using Test Sessions before they are published to the Catalog. Once they are in the catalog, users can create Flow Deployments using the Deployment Wizard or CLI. Note: The Flow Designer is a Technical Preview feature and requires an entitlement to be activated. Reach out to your Cloudera Account team to get Technical Preview access.   Added support for certificate renewal of inbound connection endpoints Six new ReadyFlows have been added to the ReadyFlow Gallery: Airtable to S3/ADLS Box to S3/ADLS Dropbox to S3/ADLS Google Drive to S3/ADLS MQTT to Kafka Salesforce to S3/ADLS DataFlow is now also available in the EU and APAC regional CDP control planes. Updated to support using kubernetes 1.23 EKS and AKS clusters. General improvements to DataFlow upgrade reliability to allow for improved success rates Includes new NiFi version 1.18.0.2.3.7.0 for flow development and flow deployment. Updated DF Function binaries to use 1.18.0 See the Flow Designer Announcement blog post and the product documentation for more information. ... View more

The latest release (2.2.0-b194) of Cloudera DataFlow (CDF) on CDP Public Cloud introduces the following new features for both, AWS and Azure customers:   A streamlined first time user experience You can use the new Hello World ReadyFlow to create your first flow deployment without dependencies on source or target systems. New landing pages for the Dashboard and Catalog allow you to get started with just one click. Four additional new ReadyFlows are available in the ReadyFlow Gallery Non-CDP ADLS to S3/ADLS Non-CDP S3 to S3/ADLS ListenTCP filter to S3/ADLS ListenSyslog filter to S3/ADLS Endpoint Hostnames certificates can now be renewed when using Inbound Connections Learn more about the new features in the CDF-PC documentation and take an interactive tour of CDF-PC. ... View more