Migrating or hosting data workloads to the cloud brings massive scalability, but without a proactive approach to cost optimisation, monthly bills can quickly spiral. Optimising your underlying cloud infrastructure, virtual machines, storage volumes, and billing models that support Cloudera Data Platform (CDP) is your first line of defense against cloud waste and a critical step toward operational excellence. Beyond cloud infrastructure cost optimisation, Cloudera Observability plays a crucial role in true cost and performance optimisation by providing deep visibility into query efficiency, job executions, and resource consumption, allowing you to identify bottlenecks, fine-tune auto-scaling, and ensure every compute cycle is utilised efficiently. Cloud Infrastructure Optimisation Your infrastructure strategy sets the foundation for efficiency. Focus on these three pillars to keep costs under control: Maximise Commercial Discount Models: Avoid running 24/7 production workloads on pure on-demand pricing. For persistent components in Production like datalake nodes, datahub nodes and for data services liftie nodes, shared services nodes and core production nodes, commit to 1-year or 3-year Reserved Instances (RIs) or Savings Plans, aiming for >85% coverage.   Recommended Practice Production Environments: Leverage Reservations and Savings Plans. Aim for >85% of compute coverage on instance families running round-the-clock. Non-Production Environments: Rely on strict Scheduling Auto shutdowns in non business hours & Periodic Right-Sizing. Clean Up Cloud Hygiene Issues: A common FinOps trap for Cloudera customers is the silent accumulation of orphaned cloud infrastructure. When a CDP cluster is terminated via the Management Console, the underlying block storage (such as AWS EBS, Azure Managed Disks, or GCP Persistent Disks) can occasionally be left behind. This risk multiplies if a cluster is "force deleted", an action that bypasses the graceful teardown sequence and strands the underlying cloud infrastructure, requiring the need for manual cleanup. Because cloud providers continue to charge for storage whether a virtual machine is attached or not, these orphaned resources act as a silent budget drain. Recommendations  Periodic Audits: Conduct regular assessments of your cloud environment to hunt down unused or leftover resources. Automated Cleanup: Implement cloud-native scripts to continuously scan for and terminate unattached volumes. Lifecycle Policies: Enforce strict retention policies to automatically delete aging or obsolete data snapshots. Adopt Modern Compute Architectures: Where supported, migrate to ARM-based instances (like AWS Graviton or Azure ARM). These modern architectures yield significant price-to-performance improvements, providing direct cost reductions and performance boosts. Cloudera Data Platform  supports AWS Graviton processors (ARM-based EC2 instances) across services like Cloudera Data Engineering (CDE) and Cloudera Operational Database (COD), delivering 15% to 20% infrastructure cost savings. AWS Graviton support for Datahub data engineering template is currently in technical preview.  For Azure, ARM-based compatibility is limited only to Impala Virtual Warehouses under Cloudera Data Warehouse (CDW). Fine-Tuning  Data Services How you configure CDP’s internal data services dictates how efficiently your cloud resources are consumed. Fine-tuning these services ensures you aren't over-provisioning compute. Data Lake & Warehousing Right-Size Data Lake Topologies: Use Enterprise Duty Data Lakes for production/staging requiring High Availability. For Dev/Test environments, deploy Light Duty Data Lakes with fewer nodes. For infrequent & adhoc data processing jobs , use entirely ephemeral environments. Optimise Cloudera Data Warehouse (CDW): Don't be too conservative with auto-scaling. Reduce auto-suspend timeouts from 60 minutes to 15 minutes and lower trigger shutdown delays for Impala coordinators to free resources faster. Tune auto suspend timeout and shutdown delays parameter value as per business levers.  Auto-suspend timeout value in the virtual warehouse will shut down the compute-heavy executor nodes when idle & immediately start scaling up compute nodes when a query is queued.. Trigger shutdown delay: This parameter controls the Coordinator. It defines how long the Impala Coordinator stays alive after the executors have been suspended. Factors to Consider When Tuning If your business users can tolerate cold start and can wait for execution of query or dashboard, keep the Auto shutdown minimum If an analyst typically spends a longer time to read the query results and then runs a follow-up query, set the timeout longer to accommodate human reading time. Batch Processing of queries, when queries are being executed in bulk & batch and it's known that no more queries will arrive for a certain duration, keep the timeout very low to kill the expensive compute. If you have heavy BI tool usage, extend the "Trigger Shutdown Delay" so the Coordinator stays alive longer. Data Engineering & Machine Learning Fine-Tune Cloudera Data Engineering (CDE): Actively use suspend/resume capabilities tied to your orchestration scheduler. For temporary testing, destroy clusters completely via code rather than letting them sit idle. Leverage Spot Instances for CDE to reduce compute costs by up to 90%.  Suspend & Resume CDE .                                                                In AWS Data engineering cluster, all infrastructure & compute node groups will be shutdown except RDS. However AWS does not natively support scaling the entire cluster down to zero,  AWS will charge a baseline hourly rate for the EKS control plane. Where as in Azure, all node groups will be scaled to "0" except DB instance, Azure Kubernetes Service (AKS) allows you to completely stop the cluster, dropping those compute charges to zero Right-Sizing & Scheduling Cloudera Machine Learning (CML): Audit CPU/GPU node groups; if utilisation stays below 40%, resize to smaller instance types. Enforce business-hours scheduling in non-production environments to avoid overnight waste. Use workbench suspend and resume capabilities to automate the scheduling of lower environments of workbench. Adjust and tune the GPU and CPU node group Auto-scale ranges, setting the minimum range to zero (where supported) to eliminate idle compute costs, and capping the maximum range based on your cloud budget and quota limits. Data-Driven Optimisation via Cloudera Observability Make data-driven sizing decisions based on actual usage rather than guesswork.  Cloudera Observability acts as a "single pane of glass" for financial governance and technical optimisation, bridging the gap between engineering efficiency and financial accountability.  Cloudera Observability monitoring capabilities "Busyness" Monitor: Utilise the Busyness percentage metric. A high busyness value indicates a cluster is consistently utilised, whereas low busyness suggests the cluster is over-provisioned. Periodically review these charts to decide whether to tune for scale up or down your compute resources.  Monitor Cluster Busyness Using Cloudera Observability. Live Query Monitoring: Instead of waiting for a query to finish to analyse its performance, monitor live running queries to catch bottlenecks (such as wait times, disk spillage, or data skew) as they happen. Real-Time Monitoring. Analyse Historical Data: Leverage historical reports to track CPU, memory, and Network I/O patterns over time to detect hotspots and capacity trends. Infrastructure Monitoring, Service Monitoring, Engine & Cluster Performance.  Storage & Data Layout Optimisation Reduce storage costs and improve performance by aligning data with storage tiers. Implement Tiering: Use "Hot vs. Cold" table analysis to identify rarely accessed data. Move cold data to cheaper storage tiers (e.g., S3 Standard-IA or Archive) to significantly lower long-term storage expenses. Cleanup Metadata Bloat: Use observability insights to identify small files and over-partitioned tables that cause NameNode stress. Consolidating these small files reduces metadata overhead and speeds up query execution. Storage & Table Analysis. Hive Metastore & Query Analysis Improve system stability and reclaim resources from inefficient usage patterns. Metastore Schema Audit: Visualise your Metastore footprint to detect bloated schemas that increase memory and CPU demand.  Metastore Analysis  Query-Level Analysis: Use the resource efficiency analysis widget to identify "heavy hitters" , the top users, queries, and resource pools consuming excessive CPU and memory. Once identified, you can apply tuning recommendations or kill rogue processes consuming capacity before they impact other users. Financial Governance Chargeback & Cost Allocation: Implement cost allocation per department or team. This allows you to measure provisioned resources against actual consumed resources, helping business units understand their financial impact. Financial Governance Cost Center allocation, Budgeting  Automated Guardrails (AutoActions): Configure AutoActions to trigger alerts for query failures or sudden resource over utilisation. Setting up budget controls and proactive alerts across environments ensures you can forecast capacity needs before the monthly bill arrives. Alerting & Auto Actions. Conclusion Optimising Infrastructure Cost is a continuous journey. By combining smart infrastructure choices, service-specific fine-tuning, and the deep visibility offered by Cloudera Observability, organisations can eliminate cloud waste and significantly improve price-to-performance. These practices don't just save money but build a more resilient, performant, and scalable data platform for the future. ... View more

Orchestration of Cloudera Spark jobs Using Argo Workflows This document aims to: Demonstrate the seamless integration of Argo Workflows with Cloudera environments. Illustrate the orchestration of Spark jobs deployed on Cloudera DataHub and Data Engineering clusters using Argo Workflows. Tools/ Components used in testing the integration: Argo Workflows v3.0+ (tested) Deployed on a Kubernetes cluster with line-of-sight connectivity to Cloudera. Cloudera CDP Public Cloud/Private Cloud Access to DataHub or Data Engineering cluster. Spark Application JAR/Python script A sample Spark job to be executed. Authentication Using Cloudera Workload credentials as Kubernetes Secrets for Data Hub & Knox token for CDE Necessary credentials for Cloudera API access. Key Concepts Argo Workflows Argo Workflows is an open-source, container-native workflow engine for orchestrating parallel jobs on Kubernetes. It operates as a Kubernetes Custom Resource Definition (CRD) and runs complex multi-step workflows. Cloudera Data Platform (CDP) CDP is an integrated data platform combining Cloudera's Data Warehousing, Machine Learning, and Engineering capabilities. Spark Jobs on Cloudera Spark jobs run on Cloudera clusters (DataHub or Data Engineering) via the Livy API on datahub and Job API URL on Cloudera Data Engineering cluster. Integration Strategy The integration uses a custom Kubernetes container in the Argo Workflow, configured to interact with the Cloudera API (e.g., Livy/ CDE job API) & use the spark-submit mechanism via the Cloudera CLI or SDK. High-Level Workflow Diagram Step 1: Argo Workflow triggers. Step 2: A Kubernetes Pod runs a container with the required Cloudera tools and API clients. Step 3: The container submits the Spark job to the target Cloudera cluster (DataHub or CDE). Step 4: The Cloudera cluster runs the Spark job. Step 5: The container polls the job status via the Cloudera API. Step 6: When the job completes (success or failure), the Argo Workflow step ends Screenshot 2026-01-30 at 11.39.05 AM.png Deployment of Argo workflows on Kubernetes  Create a namespace “Argo” Deploy the Argo Workflows Quickstart template from https://github.com/argoproj/argo-workflows/releases/latest/download/quick-start-minimal.yaml Screenshot 2026-01-30 at 11.46.07 AM.png Once deployed Argo workflow UI will be available for scheduling and orchestration of jobs: Screenshot 2026-01-30 at 11.47.14 AM.png 1. Create an Argo Workflow Template An Argo Workflow Template is a reusable, versioned Kubernetes resource defining a set of tasks (a workflow). Think of it as a function definition in programming: you define the logic once and "call" it multiple times with different parameters. A workflow template is written in YAML and has four main sections: Section Purpose Metadata The name and namespace of the template. Arguments (Parameters) Inputs like table_name, input_path, or spark_memory. Entrypoint Tells Argo which template to execute first. Templates The actual "steps"—can be a Single Container, a Script, or a DAG (Directed Acyclic Graph).   Argo workflow template has these main sections  Metadata: The "Identity Card" The metadata contains the unique identifiers assigned by Kubernetes when the job was started. name vs generateName: The generateName is the prefix (datahub-dispatch-job-). Kubernetes took that prefix and added a random suffix (8l7m7) to create the unique name. uid & resourceVersion: These are internal Kubernetes tracking IDs. You don't set these; the system generates them to manage the object's lifecycle. labels: It identifies that the Argo-server system account created this job. managedFields: This is a history log of who modified the object and when. 2. Spec: The "Instruction Manual" The spec defines how the job actually runs. This specific workflow uses a two-step hierarchy. The Entrypoint (main) Argo looks here first. It tells the system to start with the template named main. The DAG/Steps (main template)  YAML steps: - name: submit-spark-datahub template: call-livy-api  The Executioner (call api template) This is where the actual work happens. Instead of running Spark directly, it uses a lightweight Curl container to "talk" to the Cloudera cluster. image: curlimages/curl:latest: A very small Docker image that only contains the curl tool. The Command (args): It sends a POST request to the Livy API. It passes the Spark configuration (memory, script path) as a JSON payload. {{workflow.parameters.xxx}}: These are variables. Argo replaces these with the values defined at the bottom of the file (e.g., the URL and the workload credentials). Secrets (env):  YAML valueFrom: secretKeyRef: name: cloudera-creds This is a security best practice. It pulls the sensitive password from a Kubernetes Secret named cloudera-creds instead of typing it in plain text in the script. 3. Arguments: The "Inputs" At the bottom, you see the hardcoded values for this specific run: livy-api-url: The endpoint for your Spark 3 cluster. user: The Kerberos/Cloudera username. script-path: The location of your Python code in HDFS (dispatch_job_ext.py). Orchestration of Cloudera Spark Jobs Using Argo Workflows Create and Submit Spark job using Argo workflow (Datahub) Click on Create a new workflow Template, and add the Manifest file in JSON/YAML format Screenshot 2026-01-30 at 4.50.58 PM.png   2. Update or modify the Parameters, such as API URL, Username, and Job script path Screenshot 2026-01-30 at 8.46.05 PM (1).png 3. Credentials can be passed as Kubernetes secret 4. Click on Submit to trigger the Workflow Screenshot 2026-01-30 at 8.50.01 PM (1).png   Screenshot 2026-01-30 at 8.50.07 PM.png 5. Validation of the successful execution of the Spark job on Datahub Screenshot 2026-02-01 at 4.53.53 PM.png   Screenshot 2026-02-01 at 4.55.13 PM (1).png   Create and Submit Spark Job using Argo workflow (Data engineering) Creating a workflow template for Spark job execution at CDE Using knox token: https://docs.cloudera.com/runtime/7.3.1/knox-authentication/topics/security-knox-token-api.html. Screenshot 2026-02-01 at 4.42.56 PM.png 2. Submitting workflow for CDE Job creation and execution of Job Screenshot 2026-02-01 at 4.45.36 PM.png   Screenshot 2026-02-01 at 4.46.17 PM.png3. Validation of Job creation & execution on CDE  image-20260201-111741.png   Screenshot 2026-02-01 at 4.47.59 PM (1).png   Cloudera Data engineering uses Job API URL for Job submissions and CDP token for authentication..token can be generated using your workload username and password. Cloudera data hub uses livy api URL for job submissions on cluster  and workload username/ password for authentication. Sample workflow Yaml Cloudera Data Engineering. metadata: name: datahub-dispatch-job-8l7m7 generateName: datahub-dispatch-job- namespace: argo uid: 82163cb3-1ed1-4901-a49d-20feb05013eb resourceVersion: "924865" generation: 4 creationTimestamp: "2026-01-12T04:45:50Z" labels: workflows.argoproj.io/action: Update workflows.argoproj.io/actor: system-serviceaccount-argo-argo-server workflows.argoproj.io/creator: system-serviceaccount-argo-argo-server managedFields: - manager: argo operation: Update apiVersion: argoproj.io/v1alpha1 time: "2026-02-01T04:33:45Z" fieldsType: FieldsV1 fieldsV1: f:metadata: f:generateName: {} f:labels: ".": {} f:workflows.argoproj.io/action: {} f:workflows.argoproj.io/actor: {} f:workflows.argoproj.io/creator: {} f:spec: {} spec: templates: - name: main inputs: {} outputs: {} metadata: {} steps: - - name: submit-spark-datahub template: call-livy-api arguments: {} - name: call-livy-api inputs: {} outputs: {} metadata: {} container: name: "" image: curlimages/curl:latest command: - sh - -c args: - | echo "Submitting Spark job to DataHub via Proxy API..." curl -k -i -u "{{workflow.parameters.user}}:$PASSWORD" \ -X POST "{{workflow.parameters.livy-api-url}}" \ -H "Content-Type: application/json" \ -d '{ "file": "{{workflow.parameters.script-path}}", "name": "Argo_DataHub_Dispatch", "conf": { "spark.executor.memory": "2g", "spark.driver.memory": "1g" } }' env: - name: PASSWORD valueFrom: secretKeyRef: name: cloudera-creds key: password resources: {} entrypoint: main arguments: parameters: - name: livy-api-url value: https://clusterdataeng-master0.se-sandb.a465-9q4k.cloudera.site/clusterdataeng/cdp-proxy-api/livy_for_spark3/v1/batches - name: user value: sumit - name: script-path value: hdfs:///user/sumit/dispatch_job_ext.py Sample workflow YAML Cloudera Data hub metadata: name: cde-vehicle-query-ext-cde8rvw8 generateName: cde-vehicle-query-ext-cde namespace: argo uid: 9a35fe46-ebde-4b83-9983-f85b724eaea5 resourceVersion: "946692" generation: 2 creationTimestamp: "2026-02-01T04:14:14Z" labels: workflows.argoproj.io/action: Update workflows.argoproj.io/actor: system-serviceaccount-argo-argo-server workflows.argoproj.io/creator: system-serviceaccount-argo-argo-server managedFields: - manager: argo operation: Update apiVersion: argoproj.io/v1alpha1 time: "2026-02-01T10:31:22Z" fieldsType: FieldsV1 fieldsV1: f:metadata: f:generateName: {} f:labels: ".": {} f:workflows.argoproj.io/action: {} f:workflows.argoproj.io/actor: {} f:workflows.argoproj.io/creator: {} f:spec: {} spec: templates: - name: main inputs: {} outputs: {} metadata: {} container: name: "" image: curlimages/curl:latest command: - sh - -c args: - > set -e # 1. GET AUTHENTICATION TOKEN echo "--- Authenticating ---" TOKEN_URL="https://service.cde-56g8qj2f.se-sandb.a465-9q4k.cloudera.site/gateway/authtkn/knoxtoken/api/v1/token" RESP=$(curl -k -s -u "$USER:$PASS" "$TOKEN_URL") TKN=$(echo "$RESP" | sed -E 's/.*"access_token":"([^"]+)".*/\1/' | tr -d '\n') if [ -z "$TKN" ]; then echo "Token generation failed"; exit 1; fi # Define Base URLs and Names API_BASE="https://t4jsfbdj.cde-56g8qj2f.se-sandb.a465-9q4k.cloudera.site/dex/api/v1" RES_NAME="test01-res" JOB_NAME="cdedemo" # 2. ENSURE RESOURCE EXISTS echo "--- Checking Resource: $RES_NAME ---" RES_STATUS=$(curl -k -s -o /dev/null -w "%{http_code}" \ -H "Authorization: Bearer $TKN" -H "X-Requested-By: cli" \ "$API_BASE/resources/$RES_NAME") if [ "$RES_STATUS" -ne "200" ]; then echo "Creating $RES_NAME..." curl -k -s -f -X POST "$API_BASE/resources" \ -H "Authorization: Bearer $TKN" -H "X-Requested-By: cli" \ -H "Content-Type: application/json" \ -d "{\"name\": \"$RES_NAME\", \"type\": \"files\"}" fi # 3. PREPARE AND UPLOAD PYSPARK SCRIPT echo "--- Preparing and Uploading script ---" cat << 'EOF' > vehicle_query.py from pyspark.sql import SparkSession spark = SparkSession.builder.appName("VehicleQueryExt").getOrCreate() print("Executing query on vehicle_data_ext...") try: # Running the query on the updated table name df = spark.sql("SELECT * FROM default.vehicle_data_ext LIMIT 20") df.show() except Exception as e: print(f"Error: {e}") spark.stop() EOF curl -k -f -X PUT "$API_BASE/resources/$RES_NAME/vehicle_query.py" \ -H "Authorization: Bearer $TKN" -H "X-Requested-By: cli" \ -F "file=@vehicle_query.py" # 4. CREATE THE JOB (Using POST to the /jobs collection) echo "--- Ensuring Job Definition: $JOB_NAME exists ---" curl -k -X POST "$API_BASE/jobs" \ -H "Authorization: Bearer $TKN" \ -H "X-Requested-By: cli" \ -H "Content-Type: application/json" \ -d "{ \"name\": \"$JOB_NAME\", \"type\": \"spark\", \"spark\": { \"file\": \"vehicle_query.py\", \"driverMemory\": \"1g\", \"executorMemory\": \"1g\" }, \"mounts\": [ { \"resourceName\": \"$RES_NAME\" } ] }" || echo "Job already exists or created." # 5. TRIGGER THE RUN echo "--- Triggering Run for $JOB_NAME ---" curl -k -f -i -X POST "$API_BASE/jobs/$JOB_NAME/run" \ -H "Authorization: Bearer $TKN" \ -H "X-Requested-By: cli" \ -H "Content-Type: application/json" \ -d "{}" env: - name: USER valueFrom: secretKeyRef: name: cde-creds key: username - name: PASS valueFrom: secretKeyRef: name: cde-creds key: password resources: {} entrypoint: main arguments: {}   ... View more