Community Articles

Training a model from scratch is time consuming and can be expensive, which is why many teams rely on fine-tuning — take an open-weight base model that already knows English and how to reason, and teach it your domain, your tone, and your data. Parameter-efficient fine-tuning (PEFT) methods like LoRA (Low-Rank Adaptation) go one step further: freeze the base model entirely and train only a small set of extra weights — the adapters. Each adapter attaches to one of the linear layers inside a transformer block — its projections, in the language of the architecture — and works by taking the high-dimensional representation flowing through that layer, projecting it down into a much smaller space, adjusting it there, and projecting it back up. Some information is lost in the low-rank detour, but enough survives to meaningfully adapt the model, and because only the adapter parameters train, the trainable count is a small fraction of the base: a billion-parameter base ends up with roughly six m

Most enterprises treat Salesforce as a system of record for the business, and a data platform (a Lakehouse, a warehouse, a set of streaming jobs) as the place where that business data gets joined, enriched, and analyzed. The gap between the two is deceptively simple to state — "get Salesforce changes into Kafka in real time" — and surprisingly easy to get wrong. This post walks through the architecture of doing it properly: subscribing to Salesforce Change Data Capture (CDC), landing every change event in Kafka without loss, and feeding a medallion-style lakehouse downstream. It is design-focused; the code is illustrative, not a copy-paste tutorial. The problem: there is no cable you can just plug-in Salesforce publishes change events through its Pub/Sub API. This is not a REST endpoint you poll and not a Kafka broker you can point a consumer at. It is a gRPC service over HTTP/2, streaming Avro-encoded binary payloads, at api.pubsub.salesforce.com:7443. Its three RPCs matter to us: S

Purpose Demonstrate end-to-end connectivity from the latest versions of Informatica and Cloudera Data Warehouse (CDW) – Impala Virtual Warehouse & Iceberg table format using JDBC. Validate: network path, authentication, driver compatibility, and a minimal pipeline Scope In scope Establish JDBC connectivity from Informatica runtime (Secure Agent) to CDW Impala WaaS endpoint Run a minimal data pipeline Provide a repeatable runbook + troubleshooting notes Out of scope Spark execution on CDP Data Engineering / Data Hub Advanced governance/lineage/catalog integrations Production hardening (HA, DR, advanced CI/CD), unless required later Business-case functional testing Assumptions and Prerequisites Cloudera Data Warehouse (CDW) CDW environment exists and is reachable from where the Informatica Secure Agent runs. An Impala Virtual Warehouse is created and running. You have a CDW user with Ranger-RAZ permissions to: create schema/table (or write into an existing one) insert/select

Cloudera for Azure supports CMK with Azure Key Vault. This article describes how to set up a network for CDP for Azure to allow a successful Data Service deployment. There are two options to deploy Azure Key Vault access endpoints, public and private. With public deployment, Azure Key Vault FQDN is resolved into a public IP address. With private deployment, a private endpoint for Azure Key Vault must be created as well as a private DNS zone for Azure Key Vault.  Some Azure users have governance requirements to mandate a private endpoint for Azure Key Vault. CDP environment and DataHub clusters support Azure Key Vault private endpoints. However, Cloudera Data Services for Azure do not support private endpoints.  Best practice To support both Cloudera Data Services and Data Hubs, please make sure to configure the networking  configuration for the Azure Key Vault to be like the picture below. Do not choose the second and the private endpoint option. The reason will be discu

This is Article 1 in a series on sizing GPU infrastructure for foundation models. For most predictive models, GPU memory is an afterthought. You train the model, it fits, you move on. Transformer models change that. Memory becomes a primary planning concern, and the decisions that drive it are made before training starts — not discovered hours into a run. What this article offers is a straightforward way to think about how pre-training choices map to hardware requirements. Model size, sequence length, batch size, and weight representation each push memory in a specific direction. Once you can see which choice drives which cost, you can also see your levers to bring memory down, and what each of those levers costs you. This article covers training on a single GPU. Multi-GPU strategies are a separate article. Memory during a full training run comes down to four line items — plus one small fifth that exists only under mixed precision, flagged where it appears: Weights. The most strai

Frontier models have changed what's possible. Work that previously took days now takes hours. But hosted model services will not address every business need — cost, data governance, throughput ceilings, and control over model behavior all limit the ability of these services to meet all enterprise AI needs. Open-weight models — now trailing frontier capability by months rather than years — are a game changer. Fine-tuning adds real customization at a favorable total cost of ownership. And, of course, there will still be business cases that demand full model training. The optimal play is no longer a single model; it's an assortment: the right model for the right job. Model selection — and how that model is trained, customized, and deployed — has direct consequences for your hardware requirements. Considering that demand for GPU capacity routinely outpaces supply, design choices are critical. Selections such as model size, sequence length, batch size, and many more determine how much GPU m

CVE-2025-46762:Schema parsing in the parquet-avro module of Apache Parquet 1.15.0 and previous versions allows bad actors to execute arbitrary code. While 1.15.1 introduced a fix to restrict untrusted packages, the default setting of trusted packages still allows malicious classes from these packages to be executed. The exploit is only applicable if the client code of parquet-avro uses "specific" or the "reflect" model deliberately for reading Parquet files. (the "generic" model is not impacted)Recommended fix: Users are recommended to set the system property "org.apache.parquet.avro.SERIALIZABLE_PACKAGES" to an empty string on 1.15.1. This CVE is classified as Severity: 7.1 (high) External Links: CVE-2025-46762 Detail  External Control of File Name or Path  Apache Parquet Java: Potential malicious code execution from trusted packages in the parquet-avro module when reading an Avro schema from a Parquet file metadata  *NOTE: The Engineering Team is already worki

Greetings dear Apache NiFi community. I've recently had the opportunity to use NiFi to read and write JSON data to PostgresSQL using its native JSONB data type. While researching how to do this, I found very limited information, mainly just one helpful StackOverflow question and reply by  @mpayne.My use case was fairly simple: I had JSON files that I wanted to store as-is and be able to retrieve them later in the exact same form. This presented three main challenges: JSONB is a column, meaning that when writing to it,  you need to wrap up your JSON under a new field with a name matching the column. PutDatabaseRecord will not work if the data under the new wrapper field is a valid JSON, since it will try to parse it as record values. Therefore, you need to escape your JSON data before writing to the DB. When reading the data back, you need to restore the escaped JSONs back to their original form and remove the wrapper field from step 1. Writing to JSONB column

Data governance is no longer just a "nice-to-have" feature in any data platform. It is the backbone of modern analytics. For organizations using Cloudera Data Lineage (formerly Octopai), the ability to automatically map data flows across a complex landscape is a superpower. But what happens when your governance tool and your data warehouse live in completely different, isolated networks? Recently, we tackled a common but tricky architectural challenge: connecting a Windows-based Octopai client running in one AWS VPC to an Amazon Redshift Serverless cluster sitting in a separate, secured VPC. While AWS VPC Peering is the standard solution for this, we discovered that Redshift Serverless introduces some unique networking nuances—specifically around routing tables and DNS—that can trip up even experienced cloud engineers. This blog post outlines our architecture, the specific "gotchas" we encountered, and the step-by-step path to successfully extracting metadata from the standard tpch dat

Can you rephrase this title in this format: Error... occurs when... in... Spark jobs in Cloudera AI (CML) fail with 'ClassCastException: List$SerializationProxy' a JDK mismatch between driver and executor pods A JDK version mismatch between driver and executor pods Environment Component Version CDP Private Cloud Base 7.3.2 (upgraded in place from 7.3.1) Cloudera Manager 7.13.2 Cloudera Data Services 1.5.5-h2100-b23 Cloudera AI Workbench (CML) 2.0.51-h2100-b26, on OpenShift Container Platform Runtime PBJ Workbench · Python 3.11 · Standard · 2026.08 Runtime image ml-runtime-pbj-workbench-python3.11-standard:2026.08.1-b5 Spark add-on Spark 3.5.4 – CDP 7.3.1.400  (latest available; no 7.3.2 add-on offered) Table format Apache Iceberg V2, Hive external warehouse Problem In a CML session with Spark enabled, every driver-side operation succeeds: spark.sql("SHOW DATABASES").show()spark.sql("SHOW TABLES").show()df = spark.sql("SELEC

We have deployed the minor patch updating 7.1.9-1.cdh7.1.9.p1042.65851740 to 7.1.9-1.cdh7.1.9.p20000.79072338 and during the mandatory restarts, we saw Hive Metastore failing with the below error: 2026-07-23T00:18:33.533632426Z main ERROR Unable to locate appender "LOGFILE" for logger config "guru.springframework.blog.log4j2properties"WARNING: Use "yarn jar" to launch YARN applications.SLF4J: Class path contains multiple SLF4J bindings.SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p20000.79072338/jars/log4j-slf4j-impl-2.25.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p20000.79072338/jars/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]Exception in thread "main" java.lang.ExceptionInInitializerErrorCaused

Introduction This page provides a High-Level understanding of how an enterprise might leverage Qlik/Talend integrations to run big data Spark workloads on Cloudera Public Cloud CDE. A standard setup for integrations could be represented as indicated in the diagram that follows (Fig 1):CDP-PC Talend Cloud Integration Reference Primary Talend components required to showcase integration are as follows: A Trial or Enterprise account on Qlik/Talend Cloud Installation of the Talend Studio Application on the user workstation with network line of sight to CDP APIs and endpoints Installation of Talend Remote Engine in or alongside the CDP Environment with network line of sight to CDP APIs and endpoints. Download and install instructions for Studio and Remote Engine are available from the Qlik/Talend Cloud UI available after trial activation (Fig 2) Download resources for Talend Cloud Integration setup Instructions for Talend - Cloudera Data Engineering Spark  integration The follo

Migrating or hosting data workloads to the cloud brings massive scalability, but without a proactive approach to cost optimisation, monthly bills can quickly spiral. Optimising your underlying cloud infrastructure, virtual machines, storage volumes, and billing models that support Cloudera Data Platform (CDP) is your first line of defense against cloud waste and a critical step toward operational excellence. Beyond cloud infrastructure cost optimisation, Cloudera Observability plays a crucial role in true cost and performance optimisation by providing deep visibility into query efficiency, job executions, and resource consumption, allowing you to identify bottlenecks, fine-tune auto-scaling, and ensure every compute cycle is utilised efficiently. Cloud Infrastructure Optimisation Your infrastructure strategy sets the foundation for efficiency. Focus on these three pillars to keep costs under control: Maximise Commercial Discount Models: Avoid running 24/7 production workloads on pure o

I often get asked about how Cloudera counts a node, core or CCU and wanted to bring together some resources to help answer these questions.The first resource to consider is our pricing and licensing FAQ/ Updates. Please review this as it is regularly maintained as we receive inbound suggestions and provide clarifications. Pricing updateWe also provide a formal policy on metric which is captured here, this represents the formal definition of CCU, Nodes and cores. Cloudera Licensed Metrics  Cloudera Compute Unit (CCU) The link above defines in some detail what variable compute and storage is and the use of the CCU term. I will not duplicate those details here. CCU is simply a unit of measure and I will defer to the updated document as the unit of measure definition can change as pricing policies change. NodesGenerally, we determine the number of nodes based on the count returned in Cloudera Manager.Under Administration > License, the number of nodes is listed

If you have ever run Cloudera Edge Flow Manager (EFM) out of the box on Kubernetes, you know the sad little moment when the pod restarts and every agent class, every flow, and every resource you painstakingly uploaded is just… gone. EFM’s default persistence lives inside the pod’s ephemeral filesystem, and unless you wire it up properly it never sees a second day. This post is the recipe I keep coming back to for running EFM 2.3.1.0-2 on minikube with full persistence. All state survives kubectl rollout restart, minikube stop, or the laptop lid closing on you at the wrong moment. Three things need a permanent home: Metadata (agent classes, flows, agents, manifests) → PostgreSQL Agent binaries (MiNiFi C++ / Java installers) → a PVC Uploaded resources (Python scripts, JARs, custom assets) → a second PVC The third one is the piece a bare EFM install always loses on restart, and it is the reason this post exists. 1. PrerequisitesPe

Orchestration of Cloudera Spark jobs Using Argo Workflows This document aims to: Demonstrate the seamless integration of Argo Workflows with Cloudera environments. Illustrate the orchestration of Spark jobs deployed on Cloudera DataHub and Data Engineering clusters using Argo Workflows. Tools/ Components used in testing the integration: Argo Workflows v3.0+ (tested) Deployed on a Kubernetes cluster with line-of-sight connectivity to Cloudera. Cloudera CDP Public Cloud/Private Cloud Access to DataHub or Data Engineering cluster. Spark Application JAR/Python script A sample Spark job to be executed. Authentication Using Cloudera Workload credentials as Kubernetes Secrets for Data Hub & Knox token for CDE Necessary credentials for Cloudera API access. Key Concepts Argo Workflows Argo Workflows is an open-source, container-native workflow engine for orchestrating parallel jobs on Kubernetes. It operates as a Kubernetes Custom Resource Definition (C

Context In Apache HBase replication, RegionServers in the source cluster contact RegionServers in the target cluster via RPC. When security is enabled, authentication is performed at the RPC connection establishment phase using the Simple Authentication and Security Layer framework (SASL). HBase already provides the following built-in SASL authentication mechanisms: Kerberos, digest and simple. When Kerberos is enabled, credentials from the source cluster will be expected by the target cluster, which will then validate these credentials against its own KDC, using the SASL Kerberos mechanism. This relies on Kerberos GSSAPI implementation for authenticating the provided credentials against the target cluster KDC, therefore the trust for the source cluster principal must have been implemented at the Kerberos system level, by either having both cluster's credentials on the same realm, or making the target cluster KDC trust the credentials from the source cluster realm (an approach commonly

Arpit Agarwal, Shiv Moorthy, Sergio Gago Huerta 1. Executive Summary 1.1. Strategic Imperative and Market Position Cloudera Object Store is the company’s next-generation, on-premises storage solution powered by Apache Ozone. Its development is a strategic imperative for the evolving needs of enterprise data platforms, particularly for analytics, artificial intelligence (AI), and machine learning (ML) workloads. Cloudera Object Store is architected from the ground up as a scalable, distributed object store. This design choice addresses the "small files problem" and single-namespace limitation, enabling high-performance metadata operations, dense storage architectures, and support for a massive number of objects. By seamlessly integrating with the broader Big Data analytics ecosystem and offering Amazon S3 API compatibility, Ozone is a foundational enabler for modern, hybrid-cloud data lake architectures. The open-source platform's ability to run on commodity hardware provides a cos

In the GenAI era, I’ve really enjoyed how tools like LLMs can boost productivity in day-to-day operations. However, I found that NiFi’s Web UI doesn’t integrate very naturally with GenAI workflows. I also explored options like MCP Server to bridge that gap, but it requires additional setup and a specific skillset.Recently, I discovered a much simpler approach—leveraging NiFi’s REST APIs directly. Without any MCP setup, we can use LLMs as an operations co-pilot to extract, audit, and document NiFi flows and Ranger policies.What Is Vibe Coding for NiFi Development?Vibe coding means you describe your intent in natural language and let AI write the code. Applied to CDP NiFi flow development, it looks like this:You say: "Show me all Process Groups and their processor counts" AI does: Generates the correct curl commands, calls the NiFi REST API, parses the JSON, and returns a clean summary tableAs a NiFi flow developer, you spend most of your time designing data pipelines -- not memorizing A

Running Apache Airflow on Kubernetes brings incredible scalability, but when DAGs stall, tasks disappear, or the scheduler goes silent, navigating the infrastructure layer can be daunting. This guide breaks down essential debugging commands into six critical pillars, explaining why you need them, when to use them, and what to look out for during an incident. 1. Kubernetes Pods, Nodes, and Events Before digging into Airflow application logs, check the health of your underlying Kubernetes infrastructure. Pod evictions, scheduling failures, or resource constraints are often the root cause of "mysterious" task failures. List pods and states kubectl get pods -n <namespace> -o wide When to use: Your first line of defense when tasks aren't picking up or the UI is unresponsive. What to look for: Look for pods in CrashLoopBackOff, OOMKilled, or Pending states. The -o wide flag gives you the IP and node name, helping you identify if a specific Kubernetes node is failing. Iden

AI has moved from boardroom ambition to competitive necessity. Enterprises that can deploy production-grade AI on their most sensitive data, customer records, financial transactions, and patient histories will outpace those that cannot. Yet the majority of AI initiatives stall not because of a lack of talent or technology, but because of one intractable problem: how do you feed sensitive data into AI systems without violating privacy regulations, triggering compliance reviews, or opening the door to data breaches? The answer, until now, has been: you don't. You either expose data and accept the risk, or you exclude it and accept the limitations. Neither option is acceptable for a business serious about AI. Three Roadblocks Killing AI Initiatives Across industries, the same three barriers appear again and again when enterprises try to operationalize AI on sensitive data: 1. The AI-Ready Data Gap Legacy systems weren’t built for AI-scale speed and volume. You’re forced to choose: exposes

Spark Python Supportability MatrixThe Spark Python Supportability Matrix serves as an essential tool for determining which Python versions are compatible with specific Spark versions. This matrix provides a detailed overview of the compatibility levels for various Python versions across different Spark releases.Spark VersionPython Min Supported VersionPython Max Supported VersionPythonv 2.7Pythonv3.4Pythonv3.5Pythonv 3.6Pythonv3.7Pythonv3.8Pythonv3.9Pythonv3.10Pythonv3.113.5.53.83.11NoNoNoNoNoYesYesYesYes3.5.43.83.11NoNoNoNoNoYesYesYesYes3.5.33.83.11NoNoNoNoNoYesYesYesYes3.5.23.83.11NoNoNoNoNoYesYesYesYes3.5.13.83.11NoNoNoNoNoYesYesYesYes3.5.03.83.11NoNoNoNoNoYesYesYesYes3.4.23.73.11NoNoNoNoYesYesYesYesYes3.4.13.73.11NoNoNoNoYesYesYesYesYes3.4.03.73.11NoNoNoNoYesYesYesYesYes3.3.33.73.10NoNoNoNoYesYesYesYesNo3.3.23.73.10NoNoNoNoYesYesYesYesNo3.3.13.73.10NoNoNoNoYesYesYesYesNo3.3.03.73.10NoNoNoNoYesYesYesYesNo3.2.43.63.9NoNoNoYesYesYesYesNoNo3.2.33.63.9NoNoNoYesYesYesYesNoNo3.2.23.63.9No

PrerequisitesBefore beginning the key rotation process, ensure the following are in place:Virtual machine account access or the ability to create them via ssh-keygen: You must have the necessary permissions to manage Virtual Machines and Key Vaults in your cloud provider environment or via bash command line interface.New SSH key pair: Generate a new, secure SSH key pair (public and private).AssumptionsYou have an old key called id_rsa and id_rsa.pub.You have generated a new key pair, either via your cloud provider SSH Keys service or via ssh-keygen via Bash CLI. We will assume; for the purpose of this guide, that the new keys are id_rsa_new and id_rsa_new.pub.You have access to the Cloudera Control Plane. You have PowerUser or similar permissions to update the Summary tab values in your environment.1. Generate a New SSH Key PairA new, unique key pair must be generated for the rotation.Key StepCommand (Linux/macOS)DescriptionGenerate keyssh-keygen -t rsa -b 4096 -f ~/.ssh

We are all using AI to write code, but when it comes to Apache NiFi, the current landscape often resembles the Wild West. Whether you are generating synthetic data scripts or translating complex machine learning models, Large Language Models (LLMs) are incredible accelerators. However, if you ask an AI to write a native Apache NiFi 2.0 Python processor from scratch, there is a very high probability it will confidently hand you code that instantly breaks your canvas.NiFi 2.0’s Python API is relatively new, and most AI training data is heavily saturated with legacy NiFi 1.x ExecuteScript solutions (using Jython or Groovy). Even when an AI correctly identifies the 2.0 API, it frequently misconfigures the underlying Java-to-Python bridge, resulting in “ghost” processors with dashed lines and missing routing relationships.In this post, I am going to share the exact methodology I used to leverage AI for writing custom NiFi processors safely, ensuring my dataflows operate seamlessly

Optimizing Hive queries is crucial for achieving better performance and scalability in a data warehouse environment. Here are some tips and best practices for optimizing Hive queries: Partitioning: Partitioning your data can significantly improve query performance by reducing the amount of data scanned during query execution. Partition your tables based on commonly filtered columns, such as date or category. Use static partitioning for columns with a limited number of distinct values and dynamic partitioning for columns with high cardinality. Consider using partitioned tables for time-series data to improve query performance for date-range queries. Bucketing: Bucketing distributes data into a fixed number of buckets based on the hash value of one or more columns. Use bucketing to distribute data across files and improve data locality evenly. Choose the number of buckets wisely based on the size of your data and the available resources. Bucketing is particularly useful for optimi