Blogs

It’s 2:00 AM on a Friday. Maya, a Senior Machine Learning Engineer, wakes up to a piercing, relentless alarm from PagerDuty.A few hours prior, her team had successfully deployed a newly optimised version of their Customer Churn Assessment Model to production. The offline validation metrics had been stellar - higher ROC-AUC, faster feature processing, and tighter probability bounds. The team cheered, closed their laptops, and headed out for the weekend.But real-world data is a chaotic beast. Shortly after midnight, a subtle mismatch in incoming payload structures - an unexpected tensor name and strict precision specifications (FP32) enforced by the serving container - triggered widespread initialisation failures. Real-time dashboards were flashing red, inference calls were throwing HTTP errors, revenue-critical churn alerts were completely dark, and Maya had to frantically roll back the entire architecture under intense pressure.The Lesson: Traditional "all-or-nothing" cutovers are the

Enterprise data architecture has always been shaped by where data is allowed to live. Public cloud is one part of that landscape, but so are private data centers, sovereign infrastructure, edge environments, and networks that are completely isolated from the Internet. Regulatory boundaries account for some of this distribution. Data gravity, latency, existing infrastructure, security, and economics account for the rest.This creates an interesting problem for developers. The services required to build an application may be broadly familiar across environments, but the experience of consuming those services is often fragmented. Provisioning a Spark environment in one location may involve a different process from deploying it somewhere else. The tools used to inspect failures, configure access, discover data, or deploy an AI service may also change with the underlying infrastructure.Introducing Cloudera Anywhere Cloud!Cloudera Anywhere Cloud approaches this as a platform consistency probl

AI continues to reshape software engineering, and one question is becoming increasingly important: Can we trust AI agents to do more than just assist?Cloudera's VP, Engineering at Harness {unscripted}I  am excited to share that Stephen Ellis, Cloudera's VP of Engineering will be joining industry leaders at Harness Unscripted 2026 for the panel discussion:Time vs. Trust: Do You Trust Your AI Agents?Over the past year, AI has evolved from helping developers write code to taking on increasingly autonomous responsibilities across the software lifecycle. These workloads range from reviewing pull requests and investigating production issues to automating deployments and security workflows. AI agents are therefore beginning to move from suggestions to actions.As these systems become more capable, the conversation naturally shifts beyond productivity. The real question is no longer how quickly AI can help teams move, but how confidently organizations can allow AI to operate within produc

Introduction In today’s fast-paced digital landscape, the demand for real-time analytics has shifted from a luxury to a foundational requirement. Traditional batch processing, with its inherent latency and overnight data loads, is no longer sufficient to power the agility required by the modern enterprise. As organizations strive to become truly data-driven, data engineering teams face a daunting challenge: how to seamlessly integrate highly structured, heterogeneous data from multiple operational systems such as ERPs, CRMs, and BPM applications into a unified Data Lakehouse. Historically, achieving this meant building and maintaining a tangled, fragile web of individual, table-specific pipelines. This legacy approach quickly becomes a maintenance nightmare, resulting in exorbitant operational overhead and bottlenecks whenever source schemas evolve or new tables are added. To overcome this integration bottleneck, modern Data Engineering demands a consolidated, intelligent approach to s

Cloudera has a lot of ways to get a cluster. Almost none of them are “one command on your laptop.” cloudera-labs/cloudera-ce-aws is the exception: a Terraform + Ansible bundle that stands up a full Cloudera Private Cloud Community Edition cluster on AWS — Cloudera Manager, Kerberos, Auto-TLS, a real storage/compute topology — from a single ansible-navigator run. This post is me taking my freshly-released fork from zero to a running Ozone cluster, and the handful of real snags between the README and a green Cloudera Manager.Everything here is field-run against cloudera-ce-aws v1.0.0, deploying Cloudera Manager 7.13.2 / Runtime 7.3.2, from an Apple-Silicon Mac into AWS account AWS SE in us-east-2.What it actually deploysPermalinkA ring-fenced cluster — ~11 EC2 nodes — with everything a real Cloudera deployment has and a laptop demo usually fakes:Cloudera Manager with Kerberos and Auto-TLSSelf-contained DNS, Kerberos (FreeIP

IntroductionEvery Data and Platform Architect knows the nightmares: an upstream source system changes a date format without warning, a pipeline step injects corrupted records, or data freshness silently violates the SLA. Downstream dashboards display incorrect figures, AI models hallucinate on poor data—and the data engineering team spends days in reactive firefighting mode.The solution? A radical paradigm shift: Shift-Left Data Quality through the symbiosis of Data Contracts, the WAP (Write-Audit-Publish) Pattern, and Agentic Workflows - by means of Model Context Protocol (MCP) - getting context and metadata out of a governance layerHere is the blueprint for a modern, proactive data architecture that weaves governance directly into your infrastructure fabric.The Data Contract as the Single Source of TruthA Data Contract is not just a passive document:it is the API specification for your Data Product. Written as a versioned YAML or JSON file, it explicitly defines binding metadata at t

INTRODUCTION Branching and tagging concepts in Apache Iceberg were introduced in version 1.2.0.  With this feature one can carry out a lot of experiments on data without making copies and discarding them whenever they are not needed. The experiments work in isolation from production workloads without impacting production data. The concepts of tags and branches are very similar to source code version control systems like Git. As in, Git we know we can create multiple development branches on top of the production branch wherein we do our independent code changes, get it reviewed and merge them into the development branch. Later once all development is done we finally merge the branch into the production/release branch. The same concept is applicable in the case of branches created on an Iceberg table. Similarly, tag in Git refers to a specific commit which helps us to go to any release. The concept helps us to go to any specific version of the table using named references. Branching

Over the past year, we have seen an explosion of interest in AI agents. Whether it's coding assistants, research agents, or enterprise copilots, the common theme is that modern language models are increasingly capable of planning, reasoning, and invoking tools to accomplish complex tasks. But reasoning alone is not enough. Recent discussions across the industry, including work from OpenAI and Anthropic, have highlighted a recurring challenge: enterprise agents are only as good as the context available to them. Producing reliable outcomes requires much more than generating SQL or writing code. Agents need access to business metadata, schemas, relationships between datasets, organizational knowledge, and repeatable execution patterns that guide how work should be performed. This is why I often describe the problem as an enterprise context problem rather than simply an LLM problem. I recently spoke about this here.In an enterprise data platform, that context already exists in many forms.&

Enterprise AI has officially moved beyond theoretical concepts and is proving its worth in pre-production. At the recent Cloudera Partner Hackathon in Australia, four leading partner teams demonstrated how AI can solve real, complex operational problems leveraging the power of the Cloudera platform. The judging panel faced an outstanding lineup of presentations, assigning ranks to each participant based on the following criteria: Innovation & CreativityTechnical UnderstandingBusiness ImpactPresentation & PitchWhile the quality of all presentations was remarkably high, the judging panel selected Citadel Edge and Atturra as the winners based on their innovation, creativity, and the potential business impact of their solutions. These winners were invited to showcase their use cases at the ADAPT AI roundtable event with AWS held in Sydney on the 7th of May 2026.  This event was  a premier opportunity for the AI Hackathon winners to join an exclusive discussion w

If you are running NiFi Kafka or Flink based applications and workloads in kubernetes, you know that visibility is everything. You can build the most complex data pipelines in the world, but without eyes on your throughput, queues, or other streaming metrics, you’re essentially flying blind.Welcome to the ultimate page for Kubernetes native observability. In this series, we walk through the exact steps to wire up the entire Cloudera Streaming Operator architecture—NiFi, Kafka, and Flink—into a unified Prometheus and Grafana stack.By the end of this journey, you won’t just have basic health checks; you will have a single pane of glass correlating NiFi’s data flow metrics perfectly with Kafka’s topic throughput and Flink’s stream processing metrics.PrerequisitesPermalinkThis lesson assumes you have already:Completed deployment of Cloudera Streaming OperatorsHave the minikube branch of Streams Processing Hands on Lab setup completed, nifi flow is running, topics&n

I. Introduction: The Over-Provisioning ParadoxHave you ever hit "Submit" on a Spark job and felt that nagging urge to double the executor memory just in case?You’re not alone. As developers, our primary KPI is successful delivery. If a job fails at 3:00 AM because of an OutOfMemory error, that’s a direct hit to our productivity. To sleep better at night, we adopt a "Safety First" strategy: we request the maximum possible CPU and memory we think the job could ever need, or simply the maximum the cluster policy allows.But here’s the reality: those "just in case" resources usually sit idle. They are reserved, billed, and locked away from other workloads, but never actually used. When you multiply this "resource hoarding" across thousands of jobs and multi-tenant clusters, you aren't just over-budget, you're creating massive scheduling bottlenecks.At Cloudera, we realized that developers don't over-provision because they want to be wasteful; they do it because they lack visibility. You can

In this guide, I show you how I build a fully local RAG (Retrieval-Augmented Generation) system running locally on minikube with Cloudera Streaming Operators. I will show how to orchestrate Kafka, NiFi, and Qdrant to power semantic search and Q&A over documents.  The purpose is a foundation to develop use cases around RAG that will actually work on my local hardware.Let’s build: StreamToVLLM — a local RAG setup that turns your Cloudera Operator deployed cluster into a real-time, streaming-aware knowledge base. No cloud APIs. No data leaving your machine. Just pure Cloudera Streaming Operators (Kafka + NiFi) + vLLM inference + Qdrant vector search.Perfect for this GPU (RTX 4060 8 GB VRAM) setup — it comfortably runs Qwen2.5-3B-Instruct while NiFi ingests documents in real time via Kafka.We already have the Cloudera Streaming Operators stack, GPU-Accelerated Kubernetes: Setting up NVIDIA on Minikube, and some example Deploying vLLM with Qwen Ll

IntroductionAs enterprises and government organisations accelerate their transition to IPv6, platform compatibility becomes a critical requirement for mission-critical analytics systems. While IPv6 adoption is progressing rapidly, particularly in the Public Sector and other heavily regulated industries, many data platforms  including Cloudera on-premises deployments, still primarily operate on IPv4 networks.Cloudera is progressing towards full IPv6 and dual-stack support. However, current on-premises deployments are IPv4-based, and native IPv6 client access is not yet supported in production.To address immediate customer demand, especially from customers operating in IPv6-only or IPv6-preferred networks, we validated a practical interim solution; routing IPv6 client traffic through an F5 BIG-IP Virtual Edition (VE) and forwarding it to an IPv4-only Cloudera on-premises deployment running on AWS.This blog summarizes the architecture, validation outcomes, and real-world applicabilit

LLM Powered Development Using Windsurf and CAI Inference ServiceAs agencies build out their AI Factories, the biggest challenge is often integrating complex models into secure application development teams. By leveraging LLMs hosted on Cloudera AI Inference Service, organizations can work directly in IDE software development tools, such as  Windsurf, to provide AI-assisted coding, but without the risk of IP leakage, governance policy violations, and/or data sovereignty related issues.  This solution is enterprise ready, developer friendly, and provides seamless integration. WhyWith the advent of AI assisted coding tools that are powered by LLMs has completely changed the game. Organizations that are adopting the methodology to their software development practice are seeing orders of magnitude increase in productivity from their team, reduced time for features to get to market, and  increased team satisfaction.  In other words, this is a game changer, and the ab

The conversation around cloud adoption has matured significantly. It's no longer a question of if enterprises should use the cloud, but how they can strategically blend public cloud agility with the security and control of their on-premises infrastructure. This hybrid approach is now the dominant strategy for modern data-driven organisations.Cloud Modernisation Strategies - Migration vs BurstingOrganisations employ two complementary strategies for hybrid cloud adoption: workload migration to cloud  and cloud bursting. While traditional migration involves the permanent relocation of applications and datasets to the cloud for modernisation, cloud bursting dynamically extends a private data center into a public cloud. This provides temporary, on-demand compute to handle demand spikes, scaling back down as capacity needs subside.These two strategies co-exist. Migration is a long-term approach for modernising to cloud-native workloads, whereas bursting provides immediate compute e

Government agencies and commercial entities must retain data for several years and commonly experience IT challenges due to increased data volumes and new sources coming online. Due to these factors, they are starting to undergo degradation in the performance of Security Information & Event Management (SIEM’s) tools like Splunk. To continue to meet mission needs, address the increase in data sources that require protection, and manage costs, they have started research strategies that complement their SIEM investments while looking for solutions that meet or exceed their organization’s policies. The following requirements must be met to increase the performance of Splunk and maximize IT investment:Route data to utilize cost-effective data destinationsPreprocess incoming data by filtering and removing extraneous fieldsAggregate logsReplay any data Easily move workloads to Splunk Cloud Standardize the onboarding of additional data sourcesProvide observability of data pi

As enterprises continue to run critical analytics workloads on premises, developers and platform teams increasingly need deployments that go beyond service installation and focus on real workload validation. This blog shares the experience of deploying Cloudera on premises on Nutanix AHV using a step-by-step deployment guide and validating the environment with Project Axon, an end-to-end on-prem analytics use case that exercises ingestion, processing, analytics, and AI workflows across Cloudera services.Building on this foundation, the deployment was further validated by integrating Nutanix Objects as an S3-compatible object store and Nutanix Files as enterprise NFS storage for AI workloads. These additional validations demonstrate how Cloudera services can seamlessly leverage native Nutanix storage while supporting production-ready analytics, data engineering, SQL, and AI use cases.Across both validations, the goal was the same: ensure developers can build, run, and operate production

Unlocking Cross-Engine Analytics with Cloudera's Open Lakehouse Unlocking Cross-Engine Analytics: Why Open Table Formats Are Rewriting the Data PlaybookOrganizations today are trying to make one strategic move: build a data architecture that keeps them agile without forcing them into a single vendor’s ecosystem. As analytics platforms mature and hybrid cloud becomes the norm, teams want to analyze the same datasets from different engines—without endlessly copying data across environments.This is where the modern lakehouse has begun to shift. Open table formats such as Apache Iceberg, paired with platforms that embrace openness, are making “zero-copy analytics” not just possible, but practical. And when Iceberg tables are managed on Cloudera's Data & AI platform different compute engines  can operate on one authoritative dataset with consistent governance and no movement of data.Why This Matters: The End of the Data Silos EraEnterprises have spent years fighting a losing b

As enterprises modernize their on-premises data infrastructure, the combination of Cloudera Data Platform and Red Hat OpenShift Virtualization (OCP-V) is emerging as a transformative architecture.This joint engineering work demonstrates:Production-grade deployment of Cloudera on-premises Base (a.k.a Cloudera Private Cloud Base) on OCP-VSuccessful validation at 100+ OCP-V Virtualized nodesEnd-to-end integration with Data Services (CDW, CDE, CAI)Enterprise-grade security with TLS, Kerberos, LDAP, Ranger, and KnoxThis document brings together the complete deployment workflow, architecture overview, a real-world use case deployed for functional validation, along with validation results and key outcomes - serving as a practical blueprint for field, engineering, and customer teams.Why OpenShift Virtualization for Cloudera?This converged architecture encompasses the following:Unified, Cloud-Native Platform: OpenShift Virtualization enables Cloudera Base VMs and containerized workloads to run