Developer Blogs
Recently active
Cloud transformed the way we build software and as a side effect, Infrastructure became programmable. Storage became very elastic and analytics is now accessible to every team with a credit card and a dashboard.But have you as a practitioner would have noticed the data itself never became locationless. In fact, the bigger and more regulated our systems get, the more physical data becomes. It lives in countries with privacy laws, hospital firewalls and moves through factories with millisecond constraints. It is also powering defense systems that cannot touch the public internet. And increasingly, organizations are discovering a simple truth:The best place to run your data platform is often where the data already lives.And that is the architectural problem Cloudera has been built to solve: bringing the platform to the data, rather than forcing the data to move to the platform.That realization is driving a major architectural shift toward hybrid, sovereign, edge, and fully air-gapped dep
It’s 2:00 AM on a Friday. Maya, a Senior Machine Learning Engineer, wakes up to a piercing, relentless alarm from PagerDuty.A few hours prior, her team had successfully deployed a newly optimised version of their Customer Churn Assessment Model to production. The offline validation metrics had been stellar - higher ROC-AUC, faster feature processing, and tighter probability bounds. The team cheered, closed their laptops, and headed out for the weekend.But real-world data is a chaotic beast. Shortly after midnight, a subtle mismatch in incoming payload structures - an unexpected tensor name and strict precision specifications (FP32) enforced by the serving container - triggered widespread initialisation failures. Real-time dashboards were flashing red, inference calls were throwing HTTP errors, revenue-critical churn alerts were completely dark, and Maya had to frantically roll back the entire architecture under intense pressure.The Lesson: Traditional "all-or-nothing" cutovers are the
Enterprise data architecture has always been shaped by where data is allowed to live. Public cloud is one part of that landscape, but so are private data centers, sovereign infrastructure, edge environments, and networks that are completely isolated from the Internet. Regulatory boundaries account for some of this distribution. Data gravity, latency, existing infrastructure, security, and economics account for the rest.This creates an interesting problem for developers. The services required to build an application may be broadly familiar across environments, but the experience of consuming those services is often fragmented. Provisioning a Spark environment in one location may involve a different process from deploying it somewhere else. The tools used to inspect failures, configure access, discover data, or deploy an AI service may also change with the underlying infrastructure.Introducing Cloudera Anywhere Cloud!Cloudera Anywhere Cloud approaches this as a platform consistency probl
AI continues to reshape software engineering, and one question is becoming increasingly important: Can we trust AI agents to do more than just assist?Cloudera's VP, Engineering at Harness {unscripted}I am excited to share that Stephen Ellis, Cloudera's VP of Engineering will be joining industry leaders at Harness Unscripted 2026 for the panel discussion:Time vs. Trust: Do You Trust Your AI Agents?Over the past year, AI has evolved from helping developers write code to taking on increasingly autonomous responsibilities across the software lifecycle. These workloads range from reviewing pull requests and investigating production issues to automating deployments and security workflows. AI agents are therefore beginning to move from suggestions to actions.As these systems become more capable, the conversation naturally shifts beyond productivity. The real question is no longer how quickly AI can help teams move, but how confidently organizations can allow AI to operate within produc
Introduction In today’s fast-paced digital landscape, the demand for real-time analytics has shifted from a luxury to a foundational requirement. Traditional batch processing, with its inherent latency and overnight data loads, is no longer sufficient to power the agility required by the modern enterprise. As organizations strive to become truly data-driven, data engineering teams face a daunting challenge: how to seamlessly integrate highly structured, heterogeneous data from multiple operational systems such as ERPs, CRMs, and BPM applications into a unified Data Lakehouse. Historically, achieving this meant building and maintaining a tangled, fragile web of individual, table-specific pipelines. This legacy approach quickly becomes a maintenance nightmare, resulting in exorbitant operational overhead and bottlenecks whenever source schemas evolve or new tables are added. To overcome this integration bottleneck, modern Data Engineering demands a consolidated, intelligent approach to s
Cloudera has a lot of ways to get a cluster. Almost none of them are “one command on your laptop.” cloudera-labs/cloudera-ce-aws is the exception: a Terraform + Ansible bundle that stands up a full Cloudera Private Cloud Community Edition cluster on AWS — Cloudera Manager, Kerberos, Auto-TLS, a real storage/compute topology — from a single ansible-navigator run. This post is me taking my freshly-released fork from zero to a running Ozone cluster, and the handful of real snags between the README and a green Cloudera Manager.Everything here is field-run against cloudera-ce-aws v1.0.0, deploying Cloudera Manager 7.13.2 / Runtime 7.3.2, from an Apple-Silicon Mac into AWS account AWS SE in us-east-2.What it actually deploysPermalinkA ring-fenced cluster — ~11 EC2 nodes — with everything a real Cloudera deployment has and a laptop demo usually fakes:Cloudera Manager with Kerberos and Auto-TLSSelf-contained DNS, Kerberos (FreeIP
IntroductionEvery Data and Platform Architect knows the nightmares: an upstream source system changes a date format without warning, a pipeline step injects corrupted records, or data freshness silently violates the SLA. Downstream dashboards display incorrect figures, AI models hallucinate on poor data—and the data engineering team spends days in reactive firefighting mode.The solution? A radical paradigm shift: Shift-Left Data Quality through the symbiosis of Data Contracts, the WAP (Write-Audit-Publish) Pattern, and Agentic Workflows - by means of Model Context Protocol (MCP) - getting context and metadata out of a governance layerHere is the blueprint for a modern, proactive data architecture that weaves governance directly into your infrastructure fabric.The Data Contract as the Single Source of TruthA Data Contract is not just a passive document:it is the API specification for your Data Product. Written as a versioned YAML or JSON file, it explicitly defines binding metadata at t
INTRODUCTION Branching and tagging concepts in Apache Iceberg were introduced in version 1.2.0. With this feature one can carry out a lot of experiments on data without making copies and discarding them whenever they are not needed. The experiments work in isolation from production workloads without impacting production data. The concepts of tags and branches are very similar to source code version control systems like Git. As in, Git we know we can create multiple development branches on top of the production branch wherein we do our independent code changes, get it reviewed and merge them into the development branch. Later once all development is done we finally merge the branch into the production/release branch. The same concept is applicable in the case of branches created on an Iceberg table. Similarly, tag in Git refers to a specific commit which helps us to go to any release. The concept helps us to go to any specific version of the table using named references. Branching
Over the past year, we have seen an explosion of interest in AI agents. Whether it's coding assistants, research agents, or enterprise copilots, the common theme is that modern language models are increasingly capable of planning, reasoning, and invoking tools to accomplish complex tasks. But reasoning alone is not enough. Recent discussions across the industry, including work from OpenAI and Anthropic, have highlighted a recurring challenge: enterprise agents are only as good as the context available to them. Producing reliable outcomes requires much more than generating SQL or writing code. Agents need access to business metadata, schemas, relationships between datasets, organizational knowledge, and repeatable execution patterns that guide how work should be performed. This is why I often describe the problem as an enterprise context problem rather than simply an LLM problem. I recently spoke about this here.In an enterprise data platform, that context already exists in many forms.&
Enterprise AI has officially moved beyond theoretical concepts and is proving its worth in pre-production. At the recent Cloudera Partner Hackathon in Australia, four leading partner teams demonstrated how AI can solve real, complex operational problems leveraging the power of the Cloudera platform. The judging panel faced an outstanding lineup of presentations, assigning ranks to each participant based on the following criteria: Innovation & CreativityTechnical UnderstandingBusiness ImpactPresentation & PitchWhile the quality of all presentations was remarkably high, the judging panel selected Citadel Edge and Atturra as the winners based on their innovation, creativity, and the potential business impact of their solutions. These winners were invited to showcase their use cases at the ADAPT AI roundtable event with AWS held in Sydney on the 7th of May 2026. This event was a premier opportunity for the AI Hackathon winners to join an exclusive discussion w
If you are running NiFi Kafka or Flink based applications and workloads in kubernetes, you know that visibility is everything. You can build the most complex data pipelines in the world, but without eyes on your throughput, queues, or other streaming metrics, you’re essentially flying blind.Welcome to the ultimate page for Kubernetes native observability. In this series, we walk through the exact steps to wire up the entire Cloudera Streaming Operator architecture—NiFi, Kafka, and Flink—into a unified Prometheus and Grafana stack.By the end of this journey, you won’t just have basic health checks; you will have a single pane of glass correlating NiFi’s data flow metrics perfectly with Kafka’s topic throughput and Flink’s stream processing metrics.PrerequisitesPermalinkThis lesson assumes you have already:Completed deployment of Cloudera Streaming OperatorsHave the minikube branch of Streams Processing Hands on Lab setup completed, nifi flow is running, topics&n
I have been meaning to explore Apache Polaris for some time. Over the years, I have worked with a range of Apache Iceberg catalogs - Nessie, Hive Metastore, AWS Glue, Unity Catalog, and each of them reflects a different set of design trade-offs around metadata management, security, and interoperability. What makes Polaris particularly interesting to me is not just its technical direction, but the fact that it is evolving as a community-led project under the Apache Software Foundation.If you have followed my work, you will know I tend to approach systems by first understanding how they are built rather than how they are used. That is the intent here. We will examine Polaris from the inside out - its system boundaries, core building blocks, and the design principles that shape its behavior. The goal is to understand Polaris from a system design perspective so we develop the depth needed for using these systems. Let’s go!System BoundariesApache Polaris is a networked metadata servic
In this guide, I show you how I build a fully local RAG (Retrieval-Augmented Generation) system running locally on minikube with Cloudera Streaming Operators. I will show how to orchestrate Kafka, NiFi, and Qdrant to power semantic search and Q&A over documents. The purpose is a foundation to develop use cases around RAG that will actually work on my local hardware.Let’s build: StreamToVLLM — a local RAG setup that turns your Cloudera Operator deployed cluster into a real-time, streaming-aware knowledge base. No cloud APIs. No data leaving your machine. Just pure Cloudera Streaming Operators (Kafka + NiFi) + vLLM inference + Qdrant vector search.Perfect for this GPU (RTX 4060 8 GB VRAM) setup — it comfortably runs Qwen2.5-3B-Instruct while NiFi ingests documents in real time via Kafka.We already have the Cloudera Streaming Operators stack, GPU-Accelerated Kubernetes: Setting up NVIDIA on Minikube, and some example Deploying vLLM with Qwen Ll
IntroductionAs enterprises and government organisations accelerate their transition to IPv6, platform compatibility becomes a critical requirement for mission-critical analytics systems. While IPv6 adoption is progressing rapidly, particularly in the Public Sector and other heavily regulated industries, many data platforms including Cloudera on-premises deployments, still primarily operate on IPv4 networks.Cloudera is progressing towards full IPv6 and dual-stack support. However, current on-premises deployments are IPv4-based, and native IPv6 client access is not yet supported in production.To address immediate customer demand, especially from customers operating in IPv6-only or IPv6-preferred networks, we validated a practical interim solution; routing IPv6 client traffic through an F5 BIG-IP Virtual Edition (VE) and forwarding it to an IPv4-only Cloudera on-premises deployment running on AWS.This blog summarizes the architecture, validation outcomes, and real-world applicabilit
LLM Powered Development Using Windsurf and CAI Inference ServiceAs agencies build out their AI Factories, the biggest challenge is often integrating complex models into secure application development teams. By leveraging LLMs hosted on Cloudera AI Inference Service, organizations can work directly in IDE software development tools, such as Windsurf, to provide AI-assisted coding, but without the risk of IP leakage, governance policy violations, and/or data sovereignty related issues. This solution is enterprise ready, developer friendly, and provides seamless integration. WhyWith the advent of AI assisted coding tools that are powered by LLMs has completely changed the game. Organizations that are adopting the methodology to their software development practice are seeing orders of magnitude increase in productivity from their team, reduced time for features to get to market, and increased team satisfaction. In other words, this is a game changer, and the ab
Cloudera AI Code Assistant Development Using CursorAI and CAI Inference ServiceAs agencies build out their AI Factories, the biggest challenge is often integrating complex models into secure application development teams. By leveraging LLMs hosted on Cloudera AI Inference Service, organizations can work directly in IDE software development tools, such as CursorAI, to provide AI-assisted coding, but without the risk of IP leakage, governance policy violations, and/or data sovereignty related issues. This solution is enterprise-ready, developer-friendly, and provides seamless integration. WhyWith the advent of AI assisted coding tools powered by LLMs completely changing the game, Organizations that are adopting the methodology to their software development practice are seeing orders of magnitude increase in productivity from their team, reduced time for features to get to market, and increased team satisfaction. In other words, this shift is redefining
A few days back we co-hosted a Lakehouse meetup in New York City with Cloudflare and LanceDB, bringing together folks from the Apache Iceberg, Apache DataFusion, and Lance communities.At Cloudera, we view supporting and running open source meetups as critical drivers for the ecosystem itself and we have done this for a while now. Open source projects like Iceberg, DataFusion, and Lance evolve through real-world feedback, and those conversations happen best when practitioners are in the same room. These meetups create that perfect space, where design decisions are debated, trade-offs are brought in, and implementation realities are discussed openly. This event was a good example of that!We had three dedicated talks targeting these three projects and opened up the rest of the time for community networking.Iceberg Spec Evolution - v1 to v4 & how Cloudera supportsI kicked off the evening with a session on Apache Iceberg’s specification evolution. We walked through how specs v1 and v2 a
This blog is written in collaboration with Alex Merced - Head of DevRel @DremioThere was a time when talking about “Open table formats” would get you a polite nod… and then the conversation would move on. This was of course before “lakehouse” became a strategy slide in every vendor or enterprise deck. And before modular data architectures were a mainstream discussion. Back then, the idea that storage format layers should be open felt abstract and theoretical. Maybe even non-relevant in the greater scope of things!We were told, more than once:“Why does this matter?”“Isn’t Parquet enough?”“We already have a Data Warehouse & Data Lake”“Just use Delta”And to be fair, those weren’t unreasonable questions at the time. Apache Iceberg wasn’t a new query engine. It wasn’t a new database or a shiny AI model. It was basically metadata in a more technical sense. And an open table format specification. Unfortunately, these things have never been in vogue - until users started s
Government agencies and commercial entities must retain data for several years and commonly experience IT challenges due to increased data volumes and new sources coming online. Due to these factors, they are starting to undergo degradation in the performance of Security Information & Event Management (SIEM’s) tools like Splunk. To continue to meet mission needs, address the increase in data sources that require protection, and manage costs, they have started research strategies that complement their SIEM investments while looking for solutions that meet or exceed their organization’s policies. The following requirements must be met to increase the performance of Splunk and maximize IT investment:Route data to utilize cost-effective data destinationsPreprocess incoming data by filtering and removing extraneous fieldsAggregate logsReplay any data Easily move workloads to Splunk Cloud Standardize the onboarding of additional data sourcesProvide observability of data pi
As enterprises continue to run critical analytics workloads on premises, developers and platform teams increasingly need deployments that go beyond service installation and focus on real workload validation. This blog shares the experience of deploying Cloudera on premises on Nutanix AHV using a step-by-step deployment guide and validating the environment with Project Axon, an end-to-end on-prem analytics use case that exercises ingestion, processing, analytics, and AI workflows across Cloudera services.Building on this foundation, the deployment was further validated by integrating Nutanix Objects as an S3-compatible object store and Nutanix Files as enterprise NFS storage for AI workloads. These additional validations demonstrate how Cloudera services can seamlessly leverage native Nutanix storage while supporting production-ready analytics, data engineering, SQL, and AI use cases.Across both validations, the goal was the same: ensure developers can build, run, and operate production
Unlocking Cross-Engine Analytics with Cloudera's Open Lakehouse Unlocking Cross-Engine Analytics: Why Open Table Formats Are Rewriting the Data PlaybookOrganizations today are trying to make one strategic move: build a data architecture that keeps them agile without forcing them into a single vendor’s ecosystem. As analytics platforms mature and hybrid cloud becomes the norm, teams want to analyze the same datasets from different engines—without endlessly copying data across environments.This is where the modern lakehouse has begun to shift. Open table formats such as Apache Iceberg, paired with platforms that embrace openness, are making “zero-copy analytics” not just possible, but practical. And when Iceberg tables are managed on Cloudera's Data & AI platform different compute engines can operate on one authoritative dataset with consistent governance and no movement of data.Why This Matters: The End of the Data Silos EraEnterprises have spent years fighting a losing b
Open Lakehouse architecture brings in the modularity and flexibility needed to run multiple analytical workloads on top of a single source of data powered by open table formats like Apache Iceberg. Rather than relying on a tightly coupled, monolithic system, it allows teams to compose their data platform from independent building blocks - storage, table formats, catalogs, and compute engines, each selected based on specific workload and operational requirements.Storage is one of the key components of a lakehouse architecture. It is the foundation on which table formats implement transactional semantics, ACID guarantees, metadata management, and enable multi-engine interoperability. Decisions made at the storage layer directly influence how reliably tables can evolve, scale, and be shared across systems.Apache Iceberg: Database semantics on Object storageApache Iceberg implements ACID properties on top of object storage and provides a schema to refer to the data files (e.g. Apache Parqu
As enterprises modernize their on-premises data infrastructure, the combination of Cloudera Data Platform and Red Hat OpenShift Virtualization (OCP-V) is emerging as a transformative architecture.This joint engineering work demonstrates:Production-grade deployment of Cloudera on-premises Base (a.k.a Cloudera Private Cloud Base) on OCP-VSuccessful validation at 100+ OCP-V Virtualized nodesEnd-to-end integration with Data Services (CDW, CDE, CAI)Enterprise-grade security with TLS, Kerberos, LDAP, Ranger, and KnoxThis document brings together the complete deployment workflow, architecture overview, a real-world use case deployed for functional validation, along with validation results and key outcomes - serving as a practical blueprint for field, engineering, and customer teams.Why OpenShift Virtualization for Cloudera?This converged architecture encompasses the following:Unified, Cloud-Native Platform: OpenShift Virtualization enables Cloudera Base VMs and containerized workloads to run
Table of ContentsEvolution of the Apache Iceberg Table format SpecificationSpec v1: How Iceberg Enabled Analytic Tables at ScaleSpec v2: How Iceberg Supports Row-Level Deletes and Incremental MutationsSpec v3: Extended Types, Metadata Enhancements & Row LineageSpec v4: What’s Next for Developers Building with IcebergCommunity Drives InnovationCloudera’s Commitment to the Iceberg JourneyApache Iceberg began as an internal project at Netflix to solve a very real problem: analytical data lakes built on file formats like Apache Parquet and ORC lacked reliable schema evolution, atomicity, and visibility into table state. To address these challenges, the team defined a specification that could enforce consistency and transactional guarantees at scale - introducing Iceberg as the “table format for slow-moving data.”If we take a step back, most early “big data” workloads relied on the Apache Hive table format stored in the Hadoop Distributed File System (HDFS). Over time, the Hive ta
One of the most exciting and valuable features (in my opinion) of Cloudera Flow Management 4 (powered by Apache NiFi 2) is the Flow Analysis Rules engine and the various rule implementations you can bring to bear. The Flow Analysis Rules Engine is a built-in capability to Apache NiFi 2 whose mission is to help Flow Administrators enforce Best Practices in flow design. This includes limiting how many resources (threads, memory, disk, etc.) a component can leverage. For example, a Flow Admin may want to restrict all processors to use a maximum of 10 concurrent tasks. One reason for this (especially in large flows) is to prevent a Flow Designer from "hogging" the threads away from other components / teams, or to maximize parallelization across physical CPU cores rather than concurrency (which can be viewed as "apparently parallel" but not truly parallel). The same goes for disk usage that can be enforced by limiting the size of a connection's backpressure settings to prevent too much data
Already have an account? Login
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.
