Engineering Blogs

I. Introduction: The Over-Provisioning ParadoxHave you ever hit "Submit" on a Spark job and felt that nagging urge to double the executor memory just in case?You’re not alone. As developers, our primary KPI is successful delivery. If a job fails at 3:00 AM because of an OutOfMemory error, that’s a direct hit to our productivity. To sleep better at night, we adopt a "Safety First" strategy: we request the maximum possible CPU and memory we think the job could ever need, or simply the maximum the cluster policy allows.But here’s the reality: those "just in case" resources usually sit idle. They are reserved, billed, and locked away from other workloads, but never actually used. When you multiply this "resource hoarding" across thousands of jobs and multi-tenant clusters, you aren't just over-budget, you're creating massive scheduling bottlenecks.At Cloudera, we realized that developers don't over-provision because they want to be wasteful; they do it because they lack visibility. You can

The conversation around cloud adoption has matured significantly. It's no longer a question of if enterprises should use the cloud, but how they can strategically blend public cloud agility with the security and control of their on-premises infrastructure. This hybrid approach is now the dominant strategy for modern data-driven organisations.Cloud Modernisation Strategies - Migration vs BurstingOrganisations employ two complementary strategies for hybrid cloud adoption: workload migration to cloud  and cloud bursting. While traditional migration involves the permanent relocation of applications and datasets to the cloud for modernisation, cloud bursting dynamically extends a private data center into a public cloud. This provides temporary, on-demand compute to handle demand spikes, scaling back down as capacity needs subside.These two strategies co-exist. Migration is a long-term approach for modernising to cloud-native workloads, whereas bursting provides immediate compute e

Regular compaction is essential for Apache Iceberg tables to address the issue of small files, which can accumulate as transaction volume increases, leading to degraded query performance. Iceberg offers the rewrite data files Spark procedure as a mechanism for performing this compaction.What is Cloudera Lakehouse Optimizer?The Cloudera Lakehouse Optimizer (CLO) is an automated management service designed to optimize Apache Iceberg tables, thereby improving performance and reducing operational costs.It allows administrators to define maintenance policies for tables. The system evaluates these policies by gathering statistics from Iceberg metadata, which subsequently triggers maintenance actions. These actions include data compaction, snapshot expiration, and the removal of orphan files. It offers execution flexibility, supporting both schedule-based execution for predictable workloads and event-based triggers for real-time data changes.Lakehouse Optimizer maintenance actions are execute

In the world of Apache Iceberg, managing table maintenance efficiently is crucial for optimal performance and cost control. Cloudera Lakehouse Optimizer automatically manages your tables when you associate them with maintenance policies. These policies represent your maintenance intentions—defining which tables need attention and how they should be optimized.The system includes ready-to-use template policies shipped with the product, such as the ClouderaAdaptive policy. Each policy has two components: Constants (.json) and Script (.jexl), along with two execution models:Schedule Based Policies:Run maintenance during defined time windows using familiar cron expressions; must be used for predictable workloads and planned maintenance windows. A schedule-based policy must have a valid cron expression defined.Event Based Policies:Automatically trigger maintenance when actual data changes occur. Event-based policies should have a cron defined as ‘event’ rather than a standard time-based cron

The Storage Challenge Every Business FacesEnterprises create massive volumes of data—the Global Datasphere reached 149 ZB in 2024 and is projected to grow to 181 ZB by 2025 and 394 ZB by 2028. Multiple industry sources report that ~60–80% of enterprise data becomes "cold" (rarely accessed within months to a year), yet it often remains on expensive primary storage. Meanwhile, many Hadoop-based data lakes still default to 3× replication.Consider this: a company with 100TB of data actually consumes 300TB of storage due to 3x replication. What if you could automatically reduce that 300TB to under 200TB? In this blog, we'll walk through the calculations to show exactly how Cloudera's Ozone Storage Optimizer makes this possible—an intelligent system that converts cold data from expensive replication to storage-efficient Erasure Coding, reducing storage overhead from 200% to just 40–50% (≈50–53% storage reduction vs. 3× replication).Introducing Ozone St

This guide is Part 2 of our series on GRPO-based fine-tuning. It assumes you are already familiar with the core concepts of reasoning models and GRPO (see Part 1 Understanding Reasoning Models with GRPO: A Conceptual Introduction for Building your own Medical Reasoning Model ), and you are ready to apply them in practice using Python. All code is provided in a reproducible notebook, with detailed explanations and references to help you get started.Abstract: In this guide, we’ll walk step by step through fine tuning a large language model on a medical reasoning dataset from Hugging Face, using Group Relative Policy Optimization (GRPO). In the following sections, you will see exactly how the technical setup of GRPO works, how the data is structured, how to create custom reward functions, and how to test, evaluate, and save the medical reasoning model. By the end, you will have a working GRPO pipeline for building your own medical reasoning model that produces <reasoning>..</reas

1. IntroductionProblem:Most readily-available Medical AI models lack structured reasoning, which significantly limits their reliability and trustworthiness in critical clinical decision-making. GRPO offers a robust solution for building reasoning Medical AI models by reinforcing logical step-by-step explanations.Takeaway:This post explains how enterprises can fine-tune Large Language Models (LLMs)  with reinforcement learning (RL) using GRPO for more structured, explainable AI, demonstrating its application with a medical reasoning use case.2. The Business Case Why should enterprises care?Reasoning is critical in high-stakes industries like healthcare, finance, and legal sectors.  It provides better transparency, explainability, and tracing on how models generate their outputs, building crucial trust and facilitating accountability. Why GRPO?GRPO efficiently delivers advanced reasoning capabilities. It significantly reduces  compute requirements, cutting them b

Cruise Control is an open-source solution to automate the dynamic workload rebalance and self-healing of large-scale Apache Kafka clusters. With the widespread adoption of Apache Kafka, many companies now maintain Kafka clusters with hundreds or even thousands of brokers. Operating large-scale clusters may lead to frequent broker failures and a substantial burden in redistributing workload. Cruise Control was designed to solve these operational scalability challenges.What is Cloudera Streams Messaging - Kubernetes Operator? Cloudera Streams Messaging - Kubernetes Operator brings enterprise-grade Kafka deployments to existing Kubernetes infrastructures. Cloudera Streams Messaging - Kubernetes Operator allows flexible, agile, and rapid deployment as well as scaling for variable workloads. This solution enables the deployment of Kafka and related components on existing, shared Kubernetes infrastructure, eliminating the need for dedicated infrastructureCruise Control is also part of t

Contributors: Soumitra, Swami, Rishabh, Uma, Arpit Background White-box testing is essential for distributed file systems like Apache Ozone, ensuring data consistency, replication, and fault tolerance by exposing race conditions, concurrency issues, and failure scenarios. Unlike black-box testing, which focuses on system behavior without internal insights, white-box testing provides a deeper understanding of internal mechanisms, helping detect and resolve complex synchronization issues that are otherwise hard to reproduce. Why White-Box Testing for Ozone? Most downstream Ozone integration tests rely on black-box testing, which makes it difficult to automate synchronization events or reproduce escalations seen in production. White-box testing addresses this challenge by enabling precise fault injection and deterministic debugging. Early efforts used gdb to manipulate Java process threads, successfully reproducing the silent data corruption issue. However, automation was difficult with g

This story outlines the procedures undertaken to configure a multi-GPU host within the datacenter for the purpose of hosting and serving open-source models. The primary objective is to leverage two L40s GPUs to provide robust LLM inference capabilities locally. The deployment of large language models (LLMs) for local inference represents a crucial step in enhancing our on-premises AI capabilities. Given the increasing scale and complexity of contemporary LLMs, a single GPU frequently proves insufficient to accommodate the entire model within its Video RAM (VRAM). This necessitates the utilization of multiple GPUs to distribute the model's parameters and facilitate efficient inference. Our configuration, employing two L40s GPUs, is specifically designed to address this challenge, ensuring our ability to host robust LLMs that demand substantial memory and processing power for optimal performance. Prerequisites: We conducted this exercise on a host system with the following hardware and o

In the RDBMS world, query history has existed for many years. When I worked with MS SQL Server, I could easily check who executed which query on the server. In MS SQL, this feature is called Query Store. In Apache Hive and Apache Impala, the equivalent is called Query History. This feature is primarily used to simplify performance troubleshooting. With SQL queries, you can compare execution plans, check statistics, and analyze many other aspects. It gives you the ability to understand how your data warehouse is being utilized, using data from the data warehouse itself. You can even create your own reports and dashboards on top of this data. It does not stop here! You also gain insights into your query plans, compare past and present performance, and create trends based on historical data. One of the main motivations for introducing this feature in Apache Hive and Apache Impala was customer escalations in the past. There were many cases where customers complained about performance degra