Announcements

A new CLI option --switch-instance-type is added to the update-database command to switch the instance types to custom instance types for the COD clusters after the database creation. You must have the COD_USE_CUSTOM_INSTANCE_TYPES entitlement to use this option. If you plan to scale up the cluster vertically or change the existing instance type to any custom instance type, you can use the --switch-instance-type option. However, the custom instance types must be included in the allowlist by the Cloudera Operational Database. Use the list-supported-instance-types command to fetch the list of supported instance types. Following is an example of the usage. cdp opdb update-database --environment-name <env_name> --database-name <database_name> --switch-instance-type instanceGroup=INSTANCEGROUP,instanceType=instancetype For more information, see CDP CLI documentation.This is available starting in COD v1.46 and you do not need to upgrade to a higher version of

Cloudera Operational Database UI provides two new UI options Configure Database and Configure Edge Nodes to configure the worker and edge node properties in your existing operational database.You can access these menu items using the Actions menu item inside the database details page.For more information, see Configuring a database. This is available starting in COD v1.46 and you do not need to upgrade to a higher version of COD to benefit from this feature.Try the Cloudera Operational Database for free with Test Drive (no cloud account or setup is needed).

Cloudera Operational Database adds a new CLI command, list-supported-instance-types, allowing you to list the supported instance types for a combination of cloud platform, scale type, and storage type. You can filter using the instance group and architecture as well. The --custom-instance-types option in the create-database command allows you to pick and choose instance types for the COD. However, it is essential that these instance types are included in the COD allowlist. The list-supported-instance-types command is useful to fetch the list of supported instance types. Following is an example of the command. cdp opdb list-supported-instance-types --cloud-platform AZURE --storage-type CLOUD_WITH_EPHEMERAL --scale-type MICRO --instance-group WORKER --architecture X86_64For more information, see CDP CLI documentation.This is available starting in COD v1.46 and you do not need to upgrade to a higher version of COD to benefit from this feature.Try the Cloudera Operational Da

This release (2.9.0-b383) of Cloudera DataFlow on Cloudera on cloud increases developer productivity through the introduction of Parameter Groups which can be shared between flow drafts. Developers can now also create NiFi 2.0 flows in the Designer leveraging new Cloudera exclusive processors for building RAG data pipelines. Deployments can now be configured with a Prometheus endpoint that allows scraping NiFi metrics. DataFlow’s service and deployment events and alerts now support in-app and email notifications. Demo Video of release highlightsKey features for this releaseDevelop NiFi 2 flows using the Flow Designer (TP): Customers can now try out NiFi 2 easily in the Flow Designer and start building flows without having to set up new, dedicated clusters. New Cloudera exclusive GenAI processors and ReadyFlows (TP): Customers can now implement RAG pipelines by using new processors to parse, chunk and vectorize data, bringing context to their LLMs.Shared Parameter Groups: Deve

We’re pleased to announce that Cloudera Copilot for Cloudera AI Workbench is now generally available, bringing AI-powered productivity to the data science and machine learning development workflow. Cloudera Copilot is designed to accelerate development by generating code snippets, offering real-time assistance with troubleshooting, and enabling collaboration by answering questions about the project. These capabilities streamline the workflow from data exploration to model training, helping data scientists work more efficiently and reducing time spent on routine coding and debugging.Copilots have two main architectural components: the tooling and the LLM model. The tooling integrates with the developer’s IDE to deliver intelligent assistance in the coding environment, offering features like code generation, inline suggestions, and contextual guidance to streamline development. The model serves as the brain, powering these capabilities by interpreting user input and generating responses

Cloudera’s Data In Motion Team is pleased to announce the release of the Cloudera Streaming Analytics - Kubernetes Operator 1.1, an integral component of Cloudera Streaming - Kubernetes Operator. This release includes improvements to SQL Stream Builder as well as updates to Apache Flink 1.19.1. Use CasesEvent-Driven Applications: Stateful applications that ingest events from one (or more) event streams and react to incoming events by triggering computations, state updates, or external actions.Apache Flink excels in handling the concept of time and state for these applications and can scale to manage very large data volumes (up to several terabytes). It has a rich set of APIs, ranging from low-level controls to high-level functionality, like Flink SQL, enabling developers to choose the most suitable options for the implementation of advanced business logic. However, Apache Flink’s outstanding feature for event-driven applications is its support for savepoints. A savepoint is a cons

Here are some highlights from the month of September Upcoming Events     Upcoming on October 30th: ClouderaNOW: Master the Art of Delivering Trusted Data for Business Insight and Enterprise AI Check out the FY24 Cloudera Meetup Events Calendar for upcoming & past event details!   Top Activities 251 new support questions 4 new community articles 2341 new members Our Leaderboards We would like to recognize the below community members and employees for their efforts over the last month to provide community solutions.  See all our top participants at Top Solution Authors leaderboard and all the other leaderboards on our Leaderboards and Badges page.  @MattWho @shehbazk @ggangadharan @zegab @Pzahid @SAMSAL @MBockhacker @yagoaparecidoti @Crags @ren97  Share your expertise and answer some of the below open questions. Also, be sure to bookmark the unanswered question page to find additional open que

We are excited to announce the General Availability of the Cloudera AI Inference service, with NVIDIA NIM microservices in the Public Cloud. This new service enables enterprises to rapidly deploy and scale traditional, Large Language Models (LLM) and generative AI models to power AI applications. Designed with enterprise-grade security and performance optimizations, this service helps businesses unlock the full potential of AI with increased flexibility, speed, and control.As enterprises rapidly move from AI experimentation to production, the need for scalable, high-performance infrastructure becomes critical. The Cloudera AI Inference service directly addresses these needs, providing a seamless environment for deploying advanced AI models, such as LLaMA 3.1, Mistral, and Mixtral, with the ability to handle real-time inference workloads at scale. By leveraging NVIDIA NIM and high-performance GPUs enterprises can achieve up to 36x faster model inference, drastically reducing decision-ma

The Cloudera Operational Database CDP CLI adds a new option --custom-instance-types to the create-database command. Using this option, you can define custom instance types; however, the instance types must be included in the allowlist by the Operational Database.Currently, the allowlist is not accessible, however, the following are the new custom instance types supported by the Operational Database.AWS, HEAVY, HDFS:Worker: m6i.4xlarge, m7i.4xlargeMaster: m6i.8xlarge, m7i.8xlargeCompute/edge/leader: m6i.2xlarge, m7i.2xlargeGateway: r6i.8xlarge, r7i.8xlargeAWS, LIGHT, HDFS:Worker: m6i.4xlarge, m7i.4xlargeCompute/edge/leader/gateway: m6i.2xlarge, m7i.2xlargeMaster: m6i.4xlarge, m7i.4xlargeThe following is a sample output of the create-database command highlighting the usage of the --custom-instance-types option.cdp opdb create-database --environment-name cod_env --database-name cod_db --custom-instance-types masterType=m7i.4xlarge,workerType=m7i.4xlarge,leaderType=m7i.2xlarge,gatewayType=

When you create an operational database with HDFS storage type in an AWS environment, the Operational Database on HDFS clusters now also supports m6i and m7i instances for the applicable nodes. The Operational Database clusters with HDFS storage type are upgraded to enhance the operational database's performance and usability.The following are the new custom instance types supported by the Operational Database.AWS, HEAVY, HDFS:Worker: m6i.4xlarge, m7i.4xlargeMaster: m6i.8xlarge, m7i.8xlargeCompute/edge/leader: m6i.2xlarge, m7i.2xlargeGateway: r6i.8xlarge, r7i.8xlargeAWS, LIGHT, HDFS:Worker: m6i.4xlarge, m7i.4xlargeCompute/edge/leader/gateway: m6i.2xlarge, m7i.2xlargeMaster: m6i.4xlarge, m7i.4xlargeThis is available starting in COD v1.45 and you do not need to upgrade to a higher version of COD to benefit from this feature.Try the Cloudera Operational Database for free with Test Drive (no cloud account or setup is needed).

Here are some highlights from the month of August Upcoming Events     Upcoming on October 30th: ClouderaNOW: Master the Art of Delivering Trusted Data for Business Insight and Enterprise AI   Upcoming on September 26th: WEBINAR Navigating the Future of Data Management and Analytics Check out the FY24 Cloudera Meetup Events Calendar for upcoming & past event details! Top Activities 511 new support questions 1249 new members   Our Leaderboads We would like to recognize the below community members and employees for their efforts over the last month to provide community solutions.  See all our top participants at Top Solution Authors leaderboard and all the other leaderboards on our Leaderboards and Badges page.   @MattWho @vaishaakb @Scharan @vamsi_redd @vats @SAMSAL @Shelton @Juanes @yagoaparecidoti   Unanswered Questions Share your expertise and answer some of the below open q

Cloudera’s Data In Motion Team is pleased to announce the release of the Cloudera Streaming Messaging Operator 1.1, an integral component of Cloudera Streaming - Kubernetes Operator. With this release, customers receive Kafka Connect support and Kafka replication in the operator.Use CasesLoading and unloading data from Kafka: Kafka Connect gives Kafka users a simple way to get data quickly from a source and feed it to a Kafka topic. It also allows them to get data from a topic and copy it to an external destination. Adding Kafka Connect support to the operator gives our customers a tool for moving data in and out of Kafka, efficiently. Replicating data to other sites: Disaster resilience is an important aspect of any Kafka production deployment. The Cloudera Streaming Kubernetes Operator now supports configuring and running Kafka replication flows across any two Kafka clusters. These clusters could be in the same or in different data centers to provide increased resilience against

IntroductionData fuels AI. The more data that is available, the more value that can be created by AI tools leveraging that data. Cloudera Data Warehouse is an industry-leading solution for storing and managing data, and Cloudera Cloudera AI provides the tools necessary to build, train, and deploy AI models and applications. Now, those solutions work together better than ever before.Previous versions of Cloudera Data Warehouse and Cloudera Cloudera AI required duplicate authentication to integrate. Users first authenticated with Cloudera Cloudera AI and then had to re-authenticate with Cloudera Data Warehouse once they were ready to retrieve data from their warehouses. Now, Cloudera Cloudera AI seamlessly authenticates the user to Cloudera Data Warehouse without requiring the user to re-enter their credentials. In this blog, we will show how the integration between these two services works so users can leverage all of their data for AI.Let’s Get GoingThe first step is to set up an Impal

 We are pleased to announce the release of Cloudera Streaming Analytics 1.13 for Cloudera Private Cloud Base 7.1.9 SP1. This release includes improvements to SQL Stream Builder as well as updates to Apache Flink 1.19.1.Use CasesEvent-Driven Applications: Stateful applications that ingest events from one or more event streams and react to incoming events by triggering computations, state updates, or external actions.Apache Flink excels in handling the concept of time and state for these applications and can scale to manage very large data volumes (up to several terabytes) with exactly once consistency guarantees. Moreover, Apache Flink’s support for event-time, highly customizable window logic, and fine-grained control of time as provided by the ProcessFunction enable the implementation of advanced business logic. Moreover, Apache Flink features a library for Complex Event Processing (CEP) to detect patterns in data streams.However, Apache Flink’s outstanding feature for event-driv

We are thrilled to announce that Cloudera Data Engineering now supports AWS Graviton3 instances. This marks a significant milestone in our commitment to delivering high-performance, cost-effective solutions for your data engineering needs. AWS Graviton is a family of 64-bit ARM-based CPUs designed to support improved compute performance, memory bandwidth, and machine learning capabilities with less energy consumption –all essential criteria for modern data engineering workloads. This blog walks you through how you can take advantage of these powerful processors to deliver stronger results. 15%+ Cost Savings and Performance EnhancementsOne of the key benefits of switching to AWS Graviton3 instances is reducing costs, while achieving comparable performance. Cloudera customers may reduce their costs by 15-20% compared to alternative instance types.Get Started with Graviton in Cloudera Data EngineeringCreate AWS Graviton instances for Cloudera Data Engineering in six steps:Log in

We are excited to announce a series of new updates on Cloudera Data Engineering for the public cloud. Explore how you can benefit from greater performance, cost-efficiency, and versatility with new features for modern data engineering workloads.Discover New Benefits 15%+ Cost Savings with AWS Graviton: We now support AWS ARM-based processors (Graviton). Switching to Graviton instances can reduce infrastructure costs by 15-20% while achieving comparable performance. Upgraded Performance: Support for Apache Spark 3.5 and Apache Iceberg 1.4 boosts performance and provides access to more practitioner-focused capabilities, including Spark Connect and enhanced PySpark and SQL capabilities. This dual upgrade also enables multi-table commits and enhances consistency in your data operations. You can find our compatibility matrix here.Productivity Boost with Flexible Practitioner Tooling: Take advantage of iterative development with Interactive Sessions. Develop, test, and refine your

CommunityOverCode (formerly known as ApacheCon) is the official global conference series of the Apache Software Foundation (ASF). Since 1998, before the ASF’s incorporation, ApacheCon has been drawing participants at all levels to explore ”Tomorrow’s Technology Today” across 300+ Apache projects and their diverse communities. CommunityOverCode showcases the latest developments in Apache projects and emerging innovations through hands-on sessions, keynotes, real-world case studies, trainings, hackathons, and more. CommunityOverCode showcases the latest breakthroughs from ubiquitous Apache projects and upcoming innovations in the Apache Incubator, as well as open source development and leading community-driven projects the Apache way. Attendees learn about core open source technologies independent of business interests, corporate biases, or sales pitches. The CommunityOverCode program is dynamic and evolving at each event, with content directly driven by select Apache project developer a

We are excited to announce the release of the Cloudera Streaming Analytics Operator, as an integral component of Cloudera Streaming - Kubernetes Operator. The Cloudera Streaming Analytics Operator brings Apache Flink and Cloudera SQL Stream Builder to Kubernetes for our customers running Cloudera Streaming - Kubernetes Operator. This is a key release that adds strong streaming analytics capabilities to complement the Kafka features of the Operator. Cloudera SQL Stream Builder is available in Tech Preview in this release.While Kubernetes started out as a container orchestration platform where customers would mostly run stateless applications and APIs, it has now matured to a point where it can run stateful, large scale data processing frameworks such as Apache Flink and Apache Kafka. We have seen this shift in Kubernetes usage in our customer base and are excited to meet them where they are. Enabling Flink applications on Kubernetes is an important part of our vision and complements Clo

Cloudera Data Services 1.5.4 on private cloud is now generally available. This release delivers key new features across Cloudera Machine Learning, Cloudera Data Warehouse, Cloudera Data Engineering, and platform management, ensuring that customers can build cutting-edge analytics and AI solutions leveraging modern data architectures on cloud-native, on-premises infrastructure. This feature-rich release boasts 100+ new features and a multitude of enhancements, so whether you’re a data scientist, data analyst, data engineer, or platform administrator, there is plenty to be excited about.  The full details of the release are published in the release notes and documentation. In this post, I’d like to highlight three new features, and why they can be a game-changer for your business: Natural language querying in Cloudera Data Warehouse. You no longer need to wrestle with complex SQL queries and cryptic database jargon. With this innovative feature, data analysts and busi

We are excited to announce the General Availability of Cloudera Flow Management - Kubernetes Operator and Cloudera Streaming - Kubernetes Operator, allowing Cloudera customers to run Apache NiFi and Apache Kafka on Kubernetes.While Kubernetes started out as a container orchestration platform where customers would mostly run stateless applications and APIs, it has now matured to a point where it can run stateful, large scale data processing frameworks such as Apache NiFi and Apache Kafka. We have seen this shift in Kubernetes usage in our customer base and are excited to meet them where they are.Use CaseThe Kubernetes Operators allow NiFi and Kafka customers to deploy their workloads on existing shared Kubernetes clusters. Kubernetes Operators automate the installation and lifecycle management of applications on Kubernetes, which significantly lowers cost of operations and allows customers to focus on use case implementation instead of cluster operations.For customers, this means that t

 This release (2.8.0-b274) of Cloudera DataFlow (CDF) on Cloudera on cloud introduces the ability to change the flow definition version of running deployments, a new Overview page with tutorials and shortcuts, the ability to create NiFi 2.0 deployments (Technical Preview), filtering ReadyFlows by different categories, new GenAI ReadyFlows, and supports new Kubernetes versions.Key Deployment & Platform features for this releaseLatest NiFi versions:Flow Deployments now support NiFi 1.25 and NiFi 2.0Change Flow Version of existing deployments:You can now change the flow definition version of existing deployments. Changing flow versions preserves state and removes the need for tedious redeployments. Filtering & Tagging in ReadyFlow Gallery: The ReadyFlow gallery has been improved and customers can now filter for different categories, sources, destinations and compatible NiFi versions.New ‘Overview’ page: When navigating to CDF, users land in the new ‘Overview’ page which

Here are some highlights from the month of April Upcoming on May 29th: Key Considerations for GenAI in the Enterprise Check out the FY24 Cloudera Meetup Events Calendar for upcoming & past event details! 111 new support questions 5 new community articles 531 new members Rank  Community Article Author Components/ Labels #1 A Step-by-Step Guide to install Spark3 and Livy3 on Cloudera Manager with CDS3 Parcel Ranga Reddy  @RangaReddy  Apache SparkApache YARNCloudera Data Platform (CDP)Cloudera Data Platform Private Cloud (CDP-Private) #2 How to access Ozone file system using Java API Navin Kumar @Navink  Apache Ozone #3 Connect to CDP DataHub Hive using Cloudera ODBC Driver Smruti Biswal @smruti  Apache HiveCloudera Data Platform (CDP) #4 How to use Templating to make Data Engineering workloads environment agnostic with CDE Vish Rajagopalan @VishRajagopalan  Cloudera Data Engineering (CDE

In CDP CLI, while creating an operational database, you can specify the encryption key to encrypt the volume for instance groups using the --volume-encryptions (array) option. You can select this option only in AWS environments.Following is a sample command.opdb create-database --environment-name <environment-name> --database-name <database-name> --disable-external-db --scale-type MICRO --attached-storage-for-workers '{"volumeCount":1,"volumeType":"SSD","volumeSize":100}' --endpoint-url http://localhost:8988 \--volume-encryptions '[{"encryptionKey": "<aws-key-arn>","instanceGroup": "GATEWAY"}]'Shorthand syntax:encryptionKey=string,instanceGroup=string ... (separate items with spaces)JSON syntax:[{"encryptionKey": "string","instanceGroup": "WORKER"|"LEADER"|"MASTER"|"GATEWAY"|"STRONGMETA"|"EDGE"}...]For more information, see CDP CLI documentation.This is available starting in COD v1.41 and you do not need to upgrade to a higher version of COD to benefit from this featu

When you create and deploy an operational database in an Azure environment, by default, COD clusters now use Standard_D8s_v3 instance type instead of Standard_D8_v3. The instance type is upgraded to support encryption at the host level.If you want to retain the Standard_D8_v3 instance type, you must have the COD_USE_DV3_INSTANCE_TYPE entitlement on your account.This is available starting in COD v1.41 and you do not need to upgrade to a higher version of COD to benefit from this feature.Try the Cloudera Operational Database for free with Test Drive (no cloud account or setup is needed).

In CDP CLI, while creating an operational database, you can set the default root volume size with the --root-volume-size (integer) option in GiB for all the instances in the cluster.Following is a sample command.cdp opdb create-database --environment-name test-env --database-name test-db --root-volume-size 300For more information, see CDP CLI documentation.This is available starting in COD v1.41 and you do not need to upgrade to a higher version of COD to benefit from this feature.Try the Cloudera Operational Database for free with Test Drive (no cloud account or setup is needed).