What's New @ Cloudera

Accessing Apache Spark from Your Favorite IDEWorking with Apache Spark just got easier. The recent release of Cloudera Data Engineering 1.23 introduces External IDE Connectivity, powered by Spark Connect. You can now interact with Spark clusters directly from your preferred local coding environments like Jupyter Notebook, VS Code, and PyCharm. This capability helps to streamline your data engineering workflows, enabling faster development and better collaboration, all while preserving enterprise-grade security. This 6-minute demo will walk you through how to access Spark from your favorite IDE with Cloudera Data Engineering:Key BenefitsExternal IDE Connectivity brings a new level of flexibility for data engineering, allowing users to work with remote data from their local environment via secure, automated continuous integration and continuous delivery (CI/CD) pipelining. This approach offers several advantages, including:1. Develop Locally, Compute SeamlesslyDevelopers can connect

Introduction As organizations increasingly migrate their data workloads to the cloud, choosing the right storage solution is crucial. This article compares Spark3 on YARN workloads using two different Amazon S3 storage classes: S3 Standard and S3 Express One Zone running Cloudera on AWS with a 7.2.18 CDP runtime version. We'll look at the results of a series of benchmark tests highlighting the performance differences, cost implications, and key considerations for using these two storage options. Background and Motivation In cloud computing, selecting the right storage solution directly impacts the overall performance, availability, and cost of your data applications. Spark workloads that use Amazon S3 storage often face choices between multiple storage classes, each offering distinct features and trade-offs. This blog provides insight into the performance of Spark3 on YARN workloads when using S3 Standard compared to S3 Express One Zone, leveraging the industry-standard benchmarking me

Cloudera Operational Database supports setting a user as an ODAdmin at the database level. In earlier versions of the Cloudera Operational Database, you could set the ODAdmin at the environment level only, however, for better usability and enhanced security, now you can set it at the database level too. This is available starting in COD v1.48 and you do not need to upgrade to a higher version of COD to benefit from this feature.Try the Cloudera Operational Database for free with Test Drive (no cloud account or setup is needed).

Cloudera Operational Database supports creating a database with SELinux enforcement using the Cloudera Operational Database CLI. To use SELinux support, you must have the CDP_SECURITY_ENFORCING_SELINUX entitlement. Please get in touch with Cloudera support if you do not have this entitlement.The SELinux allows you to set access control through policies. You can set the SELinux mode while creating a new operational database. You can define the SELinux mode using the seLinux parameter in the create-database command. The supported SELinux modes are:ENFORCING: Enables SELinux in enforced mode, actively applying security policies.PERMISSIVE (default): Sets SELinux to permissive mode, logging any security violations without enforcing policies.If you do not define the seLinux parameter, by default, the PERMISSIVE mode is applied.The following example shows the usage of the seLinux parameter. opdb create-database --environment-name [***ENVIRONMENT_NAME***] --database-name [

We are pleased to announce the release of Cloudera Streams Messaging - Kubernetes Operator 1.2. With this release, customers receive better security integration and an update to Kafka 3.8, besides other improvements.Release HighlightsRebase on Kafka 3.8:For more information, see the Kafka 3.8 Release Notes and list of notable changes.Rebase on Strimzi 0.43.0:For more information, see the Strimzi 0.43.0 Release Notes.Apache Ranger authorization: Support for Apache Ranger authorization is now available. Customers can now integrate Kafka clusters, deployed with Cloudera Streams Messaging - Kubernetes Operator, with a remote Ranger service that is running on Cloudera Private Cloud Base. If configured, the Ranger service can provide authorization for your Kafka cluster. For more information, see Apache Ranger authorization.Improvements to Kafka replication: Rebased and backported changes to make Kafka replication more resilient and reliable when handling heartbeats and offset translation.Pe

We are pleased to announce the release of Cloudera Streaming Analytics 1.14 for Cloudera Public Cloud and Cloudera Private Cloud Base 7.3.1. This release includes improvements to SQL Stream Builder as well as updates to Apache Flink 1.19.1.Release HighlightsRebase to Apache Flink 1.19.1: Streaming analytics deployments, including SQL Stream Builder, now support Apache Flink 1.19.1, which includes the Apache Flink improvements below. For more information on these improvements and deprecations, please check the Apache Flink 1.19.1 release announcement.Custom Parallelism for Table/SQL Sources: The DataGen connector now supports setting of custom parallelism for performance tuning via the scan.parallelism option. Support for other connectors will come in future releases.Configure Different State Time to Live (TTLs) Using SQL Hint: Users have now a more flexible way to specify custom time-to-live (TTL) values for state of regular joins and group aggregations directly within t

We are thrilled to share the latest release of Cloudera Data Engineering 1.23 on public cloud. With major enhancements for developer experience and product stability, this release enables you to benefit from increased productivity and collaboration while ensuring enterprise-grade security. Discover New Benefits & FeaturesBoosted Development Productivity: The External IDE Connectivity (Tech Preview) powered by Apache Spark Connect enables practitioners to run large-scale Spark workloads remotely on Cloudera Data Engineering. That means you can access Spark from your favorite IDE and collaborate more efficiently with your team. Optimized, Cost-Effective Workflows: Support for Apache Airflow 2.9 and Python 3.11 expands access to the latest libraries and frameworks, equipping you with tools to meet your dynamic needs for building scalable data pipelines. Apache Iceberg 1.5, together with the support of Apache Spark 3.5, delivers better overall performance and cost management.

The feature to scale up the HBase REST server using the Apache HBase REST API is no longer a technical preview and is now generally available (GA). For more information, see HBase REST server scaling in COD.This is available starting in COD v1.46 and you do not need to upgrade to a higher version of COD to benefit from this feature.Try the Cloudera Operational Database for free with Test Drive (no cloud account or setup is needed).

A new CLI option --switch-instance-type is added to the update-database command to switch the instance types to custom instance types for the COD clusters after the database creation. You must have the COD_USE_CUSTOM_INSTANCE_TYPES entitlement to use this option. If you plan to scale up the cluster vertically or change the existing instance type to any custom instance type, you can use the --switch-instance-type option. However, the custom instance types must be included in the allowlist by the Cloudera Operational Database. Use the list-supported-instance-types command to fetch the list of supported instance types. Following is an example of the usage. cdp opdb update-database --environment-name <env_name> --database-name <database_name> --switch-instance-type instanceGroup=INSTANCEGROUP,instanceType=instancetype For more information, see CDP CLI documentation.This is available starting in COD v1.46 and you do not need to upgrade to a higher version of

Cloudera Operational Database UI provides two new UI options Configure Database and Configure Edge Nodes to configure the worker and edge node properties in your existing operational database.You can access these menu items using the Actions menu item inside the database details page.For more information, see Configuring a database. This is available starting in COD v1.46 and you do not need to upgrade to a higher version of COD to benefit from this feature.Try the Cloudera Operational Database for free with Test Drive (no cloud account or setup is needed).

Cloudera Operational Database adds a new CLI command, list-supported-instance-types, allowing you to list the supported instance types for a combination of cloud platform, scale type, and storage type. You can filter using the instance group and architecture as well. The --custom-instance-types option in the create-database command allows you to pick and choose instance types for the COD. However, it is essential that these instance types are included in the COD allowlist. The list-supported-instance-types command is useful to fetch the list of supported instance types. Following is an example of the command. cdp opdb list-supported-instance-types --cloud-platform AZURE --storage-type CLOUD_WITH_EPHEMERAL --scale-type MICRO --instance-group WORKER --architecture X86_64For more information, see CDP CLI documentation.This is available starting in COD v1.46 and you do not need to upgrade to a higher version of COD to benefit from this feature.Try the Cloudera Operational Da

This release (2.9.0-b383) of Cloudera DataFlow on Cloudera on cloud increases developer productivity through the introduction of Parameter Groups which can be shared between flow drafts. Developers can now also create NiFi 2.0 flows in the Designer leveraging new Cloudera exclusive processors for building RAG data pipelines. Deployments can now be configured with a Prometheus endpoint that allows scraping NiFi metrics. DataFlow’s service and deployment events and alerts now support in-app and email notifications. Demo Video of release highlightsKey features for this releaseDevelop NiFi 2 flows using the Flow Designer (TP): Customers can now try out NiFi 2 easily in the Flow Designer and start building flows without having to set up new, dedicated clusters. New Cloudera exclusive GenAI processors and ReadyFlows (TP): Customers can now implement RAG pipelines by using new processors to parse, chunk and vectorize data, bringing context to their LLMs.Shared Parameter Groups: Deve

We’re pleased to announce that Cloudera Copilot for Cloudera AI Workbench is now generally available, bringing AI-powered productivity to the data science and machine learning development workflow. Cloudera Copilot is designed to accelerate development by generating code snippets, offering real-time assistance with troubleshooting, and enabling collaboration by answering questions about the project. These capabilities streamline the workflow from data exploration to model training, helping data scientists work more efficiently and reducing time spent on routine coding and debugging.Copilots have two main architectural components: the tooling and the LLM model. The tooling integrates with the developer’s IDE to deliver intelligent assistance in the coding environment, offering features like code generation, inline suggestions, and contextual guidance to streamline development. The model serves as the brain, powering these capabilities by interpreting user input and generating responses

Cloudera’s Data In Motion Team is pleased to announce the release of the Cloudera Streaming Analytics - Kubernetes Operator 1.1, an integral component of Cloudera Streaming - Kubernetes Operator. This release includes improvements to SQL Stream Builder as well as updates to Apache Flink 1.19.1. Use CasesEvent-Driven Applications: Stateful applications that ingest events from one (or more) event streams and react to incoming events by triggering computations, state updates, or external actions.Apache Flink excels in handling the concept of time and state for these applications and can scale to manage very large data volumes (up to several terabytes). It has a rich set of APIs, ranging from low-level controls to high-level functionality, like Flink SQL, enabling developers to choose the most suitable options for the implementation of advanced business logic. However, Apache Flink’s outstanding feature for event-driven applications is its support for savepoints. A savepoint is a cons

We are excited to announce the General Availability of the Cloudera AI Inference service, with NVIDIA NIM microservices in the Public Cloud. This new service enables enterprises to rapidly deploy and scale traditional, Large Language Models (LLM) and generative AI models to power AI applications. Designed with enterprise-grade security and performance optimizations, this service helps businesses unlock the full potential of AI with increased flexibility, speed, and control.As enterprises rapidly move from AI experimentation to production, the need for scalable, high-performance infrastructure becomes critical. The Cloudera AI Inference service directly addresses these needs, providing a seamless environment for deploying advanced AI models, such as LLaMA 3.1, Mistral, and Mixtral, with the ability to handle real-time inference workloads at scale. By leveraging NVIDIA NIM and high-performance GPUs enterprises can achieve up to 36x faster model inference, drastically reducing decision-ma

The Cloudera Operational Database CDP CLI adds a new option --custom-instance-types to the create-database command. Using this option, you can define custom instance types; however, the instance types must be included in the allowlist by the Operational Database.Currently, the allowlist is not accessible, however, the following are the new custom instance types supported by the Operational Database.AWS, HEAVY, HDFS:Worker: m6i.4xlarge, m7i.4xlargeMaster: m6i.8xlarge, m7i.8xlargeCompute/edge/leader: m6i.2xlarge, m7i.2xlargeGateway: r6i.8xlarge, r7i.8xlargeAWS, LIGHT, HDFS:Worker: m6i.4xlarge, m7i.4xlargeCompute/edge/leader/gateway: m6i.2xlarge, m7i.2xlargeMaster: m6i.4xlarge, m7i.4xlargeThe following is a sample output of the create-database command highlighting the usage of the --custom-instance-types option.cdp opdb create-database --environment-name cod_env --database-name cod_db --custom-instance-types masterType=m7i.4xlarge,workerType=m7i.4xlarge,leaderType=m7i.2xlarge,gatewayType=

When you create an operational database with HDFS storage type in an AWS environment, the Operational Database on HDFS clusters now also supports m6i and m7i instances for the applicable nodes. The Operational Database clusters with HDFS storage type are upgraded to enhance the operational database's performance and usability.The following are the new custom instance types supported by the Operational Database.AWS, HEAVY, HDFS:Worker: m6i.4xlarge, m7i.4xlargeMaster: m6i.8xlarge, m7i.8xlargeCompute/edge/leader: m6i.2xlarge, m7i.2xlargeGateway: r6i.8xlarge, r7i.8xlargeAWS, LIGHT, HDFS:Worker: m6i.4xlarge, m7i.4xlargeCompute/edge/leader/gateway: m6i.2xlarge, m7i.2xlargeMaster: m6i.4xlarge, m7i.4xlargeThis is available starting in COD v1.45 and you do not need to upgrade to a higher version of COD to benefit from this feature.Try the Cloudera Operational Database for free with Test Drive (no cloud account or setup is needed).

Cloudera’s Data In Motion Team is pleased to announce the release of the Cloudera Streaming Messaging Operator 1.1, an integral component of Cloudera Streaming - Kubernetes Operator. With this release, customers receive Kafka Connect support and Kafka replication in the operator.Use CasesLoading and unloading data from Kafka: Kafka Connect gives Kafka users a simple way to get data quickly from a source and feed it to a Kafka topic. It also allows them to get data from a topic and copy it to an external destination. Adding Kafka Connect support to the operator gives our customers a tool for moving data in and out of Kafka, efficiently. Replicating data to other sites: Disaster resilience is an important aspect of any Kafka production deployment. The Cloudera Streaming Kubernetes Operator now supports configuring and running Kafka replication flows across any two Kafka clusters. These clusters could be in the same or in different data centers to provide increased resilience against

IntroductionData fuels AI. The more data that is available, the more value that can be created by AI tools leveraging that data. Cloudera Data Warehouse is an industry-leading solution for storing and managing data, and Cloudera Cloudera AI provides the tools necessary to build, train, and deploy AI models and applications. Now, those solutions work together better than ever before.Previous versions of Cloudera Data Warehouse and Cloudera Cloudera AI required duplicate authentication to integrate. Users first authenticated with Cloudera Cloudera AI and then had to re-authenticate with Cloudera Data Warehouse once they were ready to retrieve data from their warehouses. Now, Cloudera Cloudera AI seamlessly authenticates the user to Cloudera Data Warehouse without requiring the user to re-enter their credentials. In this blog, we will show how the integration between these two services works so users can leverage all of their data for AI.Let’s Get GoingThe first step is to set up an Impal

 We are pleased to announce the release of Cloudera Streaming Analytics 1.13 for Cloudera Private Cloud Base 7.1.9 SP1. This release includes improvements to SQL Stream Builder as well as updates to Apache Flink 1.19.1.Use CasesEvent-Driven Applications: Stateful applications that ingest events from one or more event streams and react to incoming events by triggering computations, state updates, or external actions.Apache Flink excels in handling the concept of time and state for these applications and can scale to manage very large data volumes (up to several terabytes) with exactly once consistency guarantees. Moreover, Apache Flink’s support for event-time, highly customizable window logic, and fine-grained control of time as provided by the ProcessFunction enable the implementation of advanced business logic. Moreover, Apache Flink features a library for Complex Event Processing (CEP) to detect patterns in data streams.However, Apache Flink’s outstanding feature for event-driv

We are thrilled to announce that Cloudera Data Engineering now supports AWS Graviton3 instances. This marks a significant milestone in our commitment to delivering high-performance, cost-effective solutions for your data engineering needs. AWS Graviton is a family of 64-bit ARM-based CPUs designed to support improved compute performance, memory bandwidth, and machine learning capabilities with less energy consumption –all essential criteria for modern data engineering workloads. This blog walks you through how you can take advantage of these powerful processors to deliver stronger results. 15%+ Cost Savings and Performance EnhancementsOne of the key benefits of switching to AWS Graviton3 instances is reducing costs, while achieving comparable performance. Cloudera customers may reduce their costs by 15-20% compared to alternative instance types.Get Started with Graviton in Cloudera Data EngineeringCreate AWS Graviton instances for Cloudera Data Engineering in six steps:Log in

We are excited to announce a series of new updates on Cloudera Data Engineering for the public cloud. Explore how you can benefit from greater performance, cost-efficiency, and versatility with new features for modern data engineering workloads.Discover New Benefits 15%+ Cost Savings with AWS Graviton: We now support AWS ARM-based processors (Graviton). Switching to Graviton instances can reduce infrastructure costs by 15-20% while achieving comparable performance. Upgraded Performance: Support for Apache Spark 3.5 and Apache Iceberg 1.4 boosts performance and provides access to more practitioner-focused capabilities, including Spark Connect and enhanced PySpark and SQL capabilities. This dual upgrade also enables multi-table commits and enhances consistency in your data operations. You can find our compatibility matrix here.Productivity Boost with Flexible Practitioner Tooling: Take advantage of iterative development with Interactive Sessions. Develop, test, and refine your

We are excited to announce the release of the Cloudera Streaming Analytics Operator, as an integral component of Cloudera Streaming - Kubernetes Operator. The Cloudera Streaming Analytics Operator brings Apache Flink and Cloudera SQL Stream Builder to Kubernetes for our customers running Cloudera Streaming - Kubernetes Operator. This is a key release that adds strong streaming analytics capabilities to complement the Kafka features of the Operator. Cloudera SQL Stream Builder is available in Tech Preview in this release.While Kubernetes started out as a container orchestration platform where customers would mostly run stateless applications and APIs, it has now matured to a point where it can run stateful, large scale data processing frameworks such as Apache Flink and Apache Kafka. We have seen this shift in Kubernetes usage in our customer base and are excited to meet them where they are. Enabling Flink applications on Kubernetes is an important part of our vision and complements Clo

Cloudera Data Services 1.5.4 on private cloud is now generally available. This release delivers key new features across Cloudera Machine Learning, Cloudera Data Warehouse, Cloudera Data Engineering, and platform management, ensuring that customers can build cutting-edge analytics and AI solutions leveraging modern data architectures on cloud-native, on-premises infrastructure. This feature-rich release boasts 100+ new features and a multitude of enhancements, so whether you’re a data scientist, data analyst, data engineer, or platform administrator, there is plenty to be excited about.  The full details of the release are published in the release notes and documentation. In this post, I’d like to highlight three new features, and why they can be a game-changer for your business: Natural language querying in Cloudera Data Warehouse. You no longer need to wrestle with complex SQL queries and cryptic database jargon. With this innovative feature, data analysts and busi

We are excited to announce the General Availability of Cloudera Flow Management - Kubernetes Operator and Cloudera Streaming - Kubernetes Operator, allowing Cloudera customers to run Apache NiFi and Apache Kafka on Kubernetes.While Kubernetes started out as a container orchestration platform where customers would mostly run stateless applications and APIs, it has now matured to a point where it can run stateful, large scale data processing frameworks such as Apache NiFi and Apache Kafka. We have seen this shift in Kubernetes usage in our customer base and are excited to meet them where they are.Use CaseThe Kubernetes Operators allow NiFi and Kafka customers to deploy their workloads on existing shared Kubernetes clusters. Kubernetes Operators automate the installation and lifecycle management of applications on Kubernetes, which significantly lowers cost of operations and allows customers to focus on use case implementation instead of cluster operations.For customers, this means that t