Community Articles

Cruise Control is an open-source solution to automate the dynamic workload rebalance and self-healing of large-scale Apache Kafka clusters. With the widespread adoption of Apache Kafka, many companies now maintain Kafka clusters with hundreds or even thousands of brokers. Operating large-scale clusters may lead to frequent broker failures and a substantial burden in redistributing workload. Cruise Control was designed to solve these operational scalability challenges.What is Cloudera Streams Messaging - Kubernetes Operator? Cloudera Streams Messaging - Kubernetes Operator brings enterprise-grade Kafka deployments to existing Kubernetes infrastructures. Cloudera Streams Messaging - Kubernetes Operator allows flexible, agile, and rapid deployment as well as scaling for variable workloads. This solution enables the deployment of Kafka and related components on existing, shared Kubernetes infrastructure, eliminating the need for dedicated infrastructureCruise Control is also part of t

What is Cloudera Global Support?Cloudera Global Support provides 24x7 customer assistance through a globally distributed team, combining deep technical expertise with proactive and predictive support capabilities to help customers successfully run Cloudera products.What issues are covered under Cloudera Support?Cloudera Support focuses on break/fix issues, including:Product crashes or failures Configuration issues Unexpected product behavior Errors during installation, upgrade, or deployment Bug fixes, patches, and workaroundsFor customer onboarding meant for new customers and/or new contacts within existing accounts to understand how Cloudera Support works, please click here.What is not covered under Cloudera Support ?Support does not include architecture design, performance tuning, training, custom solution development, or third-party system debugging. These requests are handled through Account Teams or Professional Services.How to access the support portal ?To access the support por

With the addition of Python to Apache NiFi 2.0 and Cloudera Flow Management 4, developers now have a new way to rapidly develop processors and can leverage Python libraries for FlowFile processing, which is an exciting new feature. But what about NiFi’s scripting capabilities using Jython? In this post I will explore the history of Jython and Python in the Apache NiFi project, their capabilities, and the current state of Python and Jython in both Apache NiFi 2 and Cloudera Flow Management 4. NiFi’s Scripting Capabilities: Apache NiFi 0.5.0 introduced the ExecuteScript processor, allowing developers to interact with the NiFi SDK using several JVM-based scripting languages such as Groovy, Javascript, Lua, Clojure, and Jython, a JVM-based version of Python. The Jython language is compatible with Python,  so any  script you write in pure Python should work in ExecuteScript using Jython. “Pure” Python means the entire code is written in the Python language rather than ot

Spark Streaming Graceful shutdown 1. Introduction  In the first part of the blog post, we discussed the importance of graceful shutdown in Spark Streaming applications. We also saw different ways to shutdown a Spark Streaming application gracefully, including: Explicitly calling the JVM Shutdown Hook in the driver program. Using spark.streaming.stopGracefullyOnShutdown = true setting. In the second part, we will discuss a better way to gracefully shutdown the streaming application using other approaches.   2. Use an external approach to control internal program shutdown This approach allows an external process to control the shutdown of the internal program by sending a marker record, which is detected by the internal program. The internal program can then initiate a graceful shutdown procedure, ensuring that any unsaved data is saved and any resources are properly released before exiting. Benefits: It is more reliable than relying on the internal application to shut down

Introduction:In today's digital age, data security and compliance are paramount. Organizations handling sensitive information, especially in cloud environments, need robust tools to monitor and audit user activities. Cloudera offers a comprehensive auditing system to keep track of who's doing what in your environment. In this article, we'll delve into the world of audits in CDP Cloudera, explaining why they are essential and how to use them effectively.Why Audits Matter:Enhancing Security: Audits play a pivotal role in maintaining the security of your CDP Cloudera environment. By keeping a detailed log of user activities, audits help you identify suspicious or unauthorized actions promptly.Compliance Requirements: Many industries and organizations have stringent regulatory requirements. Audits help organizations adhere to these regulations by providing a clear record of data access and modifications.Investigation and Troubleshooting: When issues arise, audits serve as valuable tools fo

Objective In this article, you will create a CDE Spark Job with dependencies in multiple CDE Repositories. On CDE and the CLI CDE is the Cloudera Data Engineering Service, a containerized managed service for Cloudera Data Platform designed for large-scale batch and Streaming Pipelines with Spark, Airflow, and Iceberg. It allows you to submit batch jobs to auto-scaling virtual clusters. As a Cloud-Native service, CDE enables you to spend more time on your applications and less on infrastructure. CDE allows you to create, manage, and schedule Apache Spark jobs without the overhead of creating and maintaining Spark clusters. With CDE, you define virtual clusters with a range of CPU and memory resources, and the cluster scales up and down as needed to run your Spark workloads, helping to control your cloud costs. CDE provides a command-line interface (CLI) client. You can use the CLI to create and update jobs, view job details, manage job resources, run jobs, etc. Requirements The followin

Introduction Data Scientists are consistently looking to augment their data sources with additional datasets either to incorporate new features, refresh the latest data, or validate existing features. In Part 1 of this two-part article, we have covered how we can ingest data using Rest API endpoints.  In Part 2, we will explore how you can add data stored in an external file system into Cloudera Data Lake.  Specifically, we will ingest data from external S3 buckets of a publicly available dataset into a Cloudera Data Lake. Data Source  We will use the the publicly available Air Quality data made available by OpenAQ for this article. To know more about OpenAQ data available for developers , you can visit their website.  Pre-requisite  Access to the Cloudera AI platform  OpenAQ API Key ( available for free, on registration at their website www.openaq.org)  We plan to use a dataset from OpenAQ data source for getting Air Quality data

Introduction Data Scientists are consistently looking to augment their data sources with additional datasets either to incorporate new features, refresh the latest data, or validate existing features. In this 2 part article, we will explore two approaches to augment external data in the Cloudera Data Lake. We will explore how REST Catalog and external S3 buckets of a publicly available dataset can be accessed within ClouderaAI to ingest data.  This is  Part 1 of a two-part article. The Part 2 covers data ingestion from an external file system (s3). Data Source  In this article, we will use the publicly available Air Quality data made available by OpenAQ. Visit their website to learn more about OpenAQ data available for developers.  Pre-requisite  Access to the Cloudera AI platform  OpenAQ API Key ( available for free, on registration at their website www.openaq.org)  We plan to use a dataset from OpenAQ data source for getting Air Quality data for ci

Objective Deploying Dask Cuda Clusters on Kubernetes for distributed GPU workloads can require time, effort, and money, especially at Enterprise scale. You'd have to set up a Kubernetes cluster with GPU support, manage docker images, and dask cuda workers via complex deployment files. Cloudera AI simplifies distributed compute use cases in the context of Machine Learning and AI. In this article, you will learn how to quickly deploy a Dask Cuda Cluster in Cloudera AI using the cmlextensions library. Background Information Dask CUDA clusters enable scalable parallel computing on NVIDIA GPUs by leveraging Dask, a flexible parallel computing framework, in conjunction with the power of CUDA for GPU acceleration. These clusters allow users to distribute computation across multiple GPUs, significantly speeding up data processing and machine learning workflows, particularly for tasks involving large datasets or complex algorithms. Cloudera AI is a suite of artificial intelligence and machine l

Hive-Kafka Integration stepsThis procedure is well suited for a Kerberos and auto TLS-enabled cluster.To begin with please create client.properties files with entries as shown below in the bootstrap server(Active Controller Node check this from cm-->kafka--->instances-->xxxx.net.com(activecontroller):security.protocol=SASL_SSLsasl.mechanism=GSSAPIsasl.kerberos.service.name=kafkassl.truststore.location=/var/run/cloudera-scm-agent/process/71-kafka-KAFKA_CONNECT/cm-auto-global_truststore.jksssl.truststore.password=xxxxxxxssl.keystore.location=/var/run/cloudera-scm-agent/process/71-kafka-KAFKA_CONNECT/cm-auto-host_keystore.jksssl.keystore.password=xxxxxxxsasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true;2. Also it is important to export the Jaas.conf file from the respective path as given below.export KAFKA_OPTS="-Djava.security.auth.login.config=/tmp/jaas.conf"Topic creation in Kafka (please get a valid tkt if it is a Kerberos-enabled clus

In YARN, resource allocation discrepancies can occur due to the way resource calculation is handled. By default, resource availability is determined based on available memory.However, when CPU scheduling is enabled, resource calculation considers both available memory and vCores. As a result, in some scenarios, nodes may appear to allocate more vCores than the configured limit while simultaneously displaying lower available resources.This happens due to the way YARN dynamically assigns vCores based on workload demands rather than strictly adhering to preconfigured limits. Additionally, in cases where CPU scheduling is disabled, YARN relies solely on memory-based resource calculation. This may lead to negative values appearing in the YARN UI, which can be safely ignored, as they do not impact actual resource utilization.

Introduction This post explains the process of submitting Spark jobs on CDP Public Cloud Data Hub to the Livy server and provides a sample wrapper script for job submission. To understand the Livy Spark submit process, please refer to this post and understand how to arrive at the job configuration. The following steps submit the Spark Pi job from the spark_examples*.jar on S3. Steps Get the Livy endpoint for Data hub from the CDP control plane Ensure that you have the workload password set for the environment Download the Python wrapper script using the following command: wget https://raw.githubusercontent.com/karthikeyanvijay/cdp-publiccloud/master/datahub-scripts/livy-cdp-spark-submit/cdp_spark_submit.py Edit the script to modify the Livy endpoint, workload username, and password Copy the spark sample jar to the S3 hdfs dfs -cp /opt/cloudera/parcels/CDH-7.2.7-1.cdh7.2.7.p6.11615609/lib/spark/examples/jars/spark-examples_2.11-2.4.5.7.2.7.6-2.jar \ s3a://vkarthikeyan/ &

Cloudera on cloud offers a scalable, flexible, and cost-efficient solution for data management and analytics in the public cloud. It integrates data applications into a unified platform, enabling seamless data consolidation, advanced analytics, and machine learning. With its pay-as-you-go pricing model and managed services, Cloudera on cloud optimizes costs and simplifies operations. It enhances security and compliance, supports hybrid and multi-cloud environments, and provides tools for real-time data processing and self-service analytics. Cloudera on cloud empowers organizations to manage data efficiently while leveraging the latest cloud technologies.In comparison to Cloudera on premises, costs in public cloud can escalate quickly if not carefully monitored, a common challenge with all cloud workloads. However, with effective tracking and cost optimization strategies, public cloud adoptions can offer significant benefits in performance, scalability and cost savings over traditional

Pre-requisitesProcessStep 1: Install the [Impala ODBC Connector 2.7.0 for Cloudera Enterprise]Step 2: Copy the ODBC folder to Power BI Desktop folderStep 3: Edit the config fileStep 4: Run Power BI DesktopStep 5: Test Connection using ODBCStep 6: Fetch data in Power BI Desktop Pre-requisites Power BI Desktop Edition  Impala in CDP Private Cloud Base Impala ODBC Connector 2.7.0 for Cloudera Enterprise CDP Public Cloud Datahub + Kerberos + Power BI Desktop  (in the future) Process Step 1: Install the [Impala ODBC Connector 2.7.0 for Cloudera Enterprise]  Step 2: Copy the ODBC folder to Power BI Desktop folder Assume your Power BI Desktop is in [ C:\Program Files\Microsoft Power BI Desktop\ ], then please copy the ODBC Driver [ C:\Program Files\Cloudera ODBC Driver for Impala ] to [C:\Program Files\Microsoft Power BI Desktop\bin\ODBC Drivers\Cloudera ODBC Driver for Impala] Step 3: Edit the config file [ C:\Program Files\Microsoft Power BI Desktop

In this tutorial, we will learn how to create Apache Ozone volumes, buckets, and keys. After that, we will see how we can access Apache Ozone data in Apache Spark. Ozone Create the volume with the name vol1 in Apache Ozone.# ozone sh volume create /vol121/08/25 06:23:27 INFO rpc.RpcClient: Creating Volume: vol1, with root as owner. Create the bucket with the name bucket1 under vol1.# ozone sh bucket create /vol1/bucket121/08/25 06:24:09 INFO rpc.RpcClient: Creating Bucket: vol1/bucket1, with Versioning false and Storage Type set to DISK and Encryption set to false Create the employee.csv file to upload to Ozone.# vi /tmp/employee.csvid,name,age1,Ranga,332,Nishanth,43,Raja,60 Upload the employee.csv file to Ozone# ozone sh key put /vol1/bucket1/employee.csv /tmp/employee.csv Add the fs.o3fs.impl property to core-site.xml Go to Cloudera Manager > HDFS > Configuration > search for core-site.xml > Cluster-wide Advanced Configuration Snippet

Objective This project provides a quickstart for submitting Spark applications compiled from Scala code in Cloudera Data Engineering. It contains useful information targeted towards Spark developers and users who are migrating to Cloudera Data Engineering, such as: Context about Scala Jobs and the usage of Jars when running Spark Applications in CDE. Useful information about the CDE CLI and how it can simplify your migration efforts to CDE. Links to references and other recommended examples, such as submitting Spark App Code located in S3 and the CDE Spark Migration CLI Tool. Requirements To reproduce this example, you will need a CDE Service in Cloudera on Cloud or On Prem and a working installation of the CDE CLI. The Scala job application has already been compiled into a Jar via sbt and is available in the `target/scala-2.12/cde-scala-example_2.12-0.1.jar` path. Background Information on Scala, Spark and the Jar File Usage A JAR (Java ARchive) file is essentially a compressed file

ObjectiveIn this brief example you will learn how to use the CDE CLI to create a CDE Spark Job with PySpark and Scala app code located in an S3 bucket.RequirementsIn order to reproduce these examples in your CDE Virtual Cluster you will need:A Spark application in the form of a PySpark script or Scala jar, located in an S3 bucket.A working installation of the CDE CLI.A CDE 1.23 Service and a Spark 3 Virtual Cluster running in Cloudera on Cloud or Cloudera on Prem.Using Dependencies located in S3 with the Spark-SubmitCDE provides a command line interface (CLI) client. You can use the CLI to create and update jobs, view job details, manage job resources, run jobs, etc.Apache Spark Spark-Submit allows you to run a Spark job with application code located in an S3 bucket. The CDE CLI also provides this functionality.For example, in PySpark:  spark-submit \ --master yarn \ --deploy-mode cluster \ --py-files s3://your-bucket/path/to/dependency_one.zip, s3://your-bucket/path/to/de

Installing and Exploring Spark 2.0 with Jupyter Notebook and Anaconda Python in your laptop 1-Objective 2-Installing Anaconda Python 3-Checking Python Install 4-Installing Spark 5-Checking Spark Install 6-Launching Jupyter Notebook with PySpark 2.0.2 7-Exploring PySpark 2.0.2 a.Spark Session b.Read CSV i.Spark 2.0 and Spark 1.6 ii.Pandas c.Pandas DataFrames, Spark DataSets, DataFrames and RDDs d.Machine Learning Pipeline i.SciKit Learn ii.Spark MLLib, ML 8-Conclusion 1-Objective It is often useful to have python with the Jupyter notebook installed on your laptop in order to quickly develop and test some code ideas or to explore some data. Adding the ability to combine Apache Spark to this will also allow you to prototype ideas and exploratory data pipelines before hitting a Hadoop cluster and paying for Amazon Web Services. We leverage the power of the Python ecosystem with libraries such as Numpy (scientific computing library of high-level mathematical functions to operate

While generative AI dominates today's headlines, traditional predictive machine learning models continue to drive critical business decisions across industries. To ensure predictive models achieve a solid ROI, well after models are initially deployed,  establishing a Machine Learning Operations (MLOps) plan is essential. MLOps is the practice of streamlining the entire lifecycle of machine learning models—from development and training to deployment, monitoring, and maintenance—in a repeatable, scalable, and governable way. Think of it as bringing software engineering discipline to machine learning, ensuring that your AI investments don't remain theoretical exercises but become dependable business assets that continue to deliver value over time. Without robust MLOps practices, models often degrade in production as data shifts over time. What begins as an impressive prototype can quickly become unreliable, leading to poor decision quality with real financial consequences. Poor model

OverviewTPC-DS (Transaction Processing Performance Council - Decision Support) is a benchmark suite for evaluating the performance of data warehouse and decision support workloads. This guide provides a step-by-step approach to setting up and executing TPC-DS on Cloudera Datahub Spark Cluster, covering data generation, query execution, and performance analysis.Note: Required files for implementation can be found in the attached GitHub repository:👉tpcds-cloudera GitHub RepositoryDatagenThis document provides step-by-step instructions to set up and run TPC-DS Data Generation on a Cloudera Datahub Spark Cluster.Step 1: Create spark-sql-perf DirectoryCreate the directory on the master node:The directory structure and name are not mandatory, but ensure to update the commands accordingly if changed.Step 2: Setup Storage Bucket DirectoryCreate the TPC-DS directory inside the data bucket:Step 3: Copy the Datagen JAR to Master NodeCopy the spark-tpcds-datagen-2.12.jar from the local system to

CREATING A CDE REPOSITORY WITH A PRIVATE GIT REPOSITORY OBJECTIVE In this article, you will learn how to create a CDE Repository by cloning a private Git repository using CDE Credentials. Cloudera Data Engineering is a cloud-based service that helps businesses build, manage, and scale data pipelines. It is based on Apache Spark and integrates with other Cloudera services. Git repositories allow teams to collaborate, manage project artifacts, and promote applications from lower to higher environments. CDE Repositories can be used to clone git repositories into a Virtual Cluster to make the imported files available to CDE Jobs as reusable dependencies. CDE currently supports Git providers such as GitHub, GitLab, and Bitbucket. CDE Credentials allow you to store secrets such as usernames, passwords, and tokens, in secure, reusable objects associated with CDE Jobs, Custom Runtimes and Repositories. REQUIREMENTS This example was built with CDE 1.23. To reproduce the commands in your environ

OBJECTIVE:Step-by step instructions for enabling the git persistence provider for NiFi in Cloudera Flow Management (CFM) on Cloudera Manager. OVERVIEW:The default install of early releases of Cloudera Flow Management (CFM) includes only basic configurations for NiFi Registry in Cloudera Manager.This article explains how to reconfigure the Nifi Registry persistence providers to use git instead of the filesystem provider. PREREQUISITES:CFM (CDF) 1.0.1 or later and CM 5.16 or CM 6.2 or later ADDING NiFi METRICS TO CM:1. Pick a location where the Nifi Registry will store local git repository files. This location has to be accessible to the NiFi Registry service account (typically: nifiregistry). It must also be initialized as a git repository using git init and linked to a remote repository using git remote add origin. (Alternatively, an existing git repository can be cloned to create the folder using git clone). The substitution variable ${nifi.registry.working.directory} c

 SummaryThis article details the resolution of a startup failure for Knox Gateway roles during a Data Hub upgrade, involving the reconfiguration of a missing knox_pac4j_password.SymptomsWhile upgrading the Data Hub cluster, Knox Gateway services failed to start.Error message: java.lang.IllegalArgumentException: The variable [${knox_pac4j_password}] does not have a corresponding value.The upgrade process was halted due to this failure.CauseThe probable cause of the startup failure was an improperly generated or missing knox_pac4j_password during the upgrade process.The built-in upgrade handler should have automatically generated the Knox secret, but it did not.InstructionsTo resolve the issue, the following steps were taken:Verify if an entry for knox_pac4j_password exists in the Cloudera Manager database.Execute SQL query: SELECT * FROM configs WHERE attr LIKE '%knox_pac4j_password%';Generate a 16-bit UUID random password.Use OpenSSL: openssl ran

SummaryResolved service errors on Datanode#09 by identifying and addressing NFS mount issues. The Cloudera agent was reconfigured to bypass NFS checks temporarily, allowing services to return to a healthy state.SymptomsMultiple service errors reported on Datanode#09.Cloudera agent was not in contact with Cloudera Manager.Filesystem usage for certain nodev filesystems was unknown due to an inactive worker process.CauseThe NFS partition was not properly mounted on the host, which led to problems with the Cloudera agent's health checks.The Cloudera agent configuration was set to monitor NFS mounts (monitored_nodev_filesystem_types=nfs,nfs4,tmpfs), which failed due to the unmounted NFS partition.InstructionsTemporarily comment out the NFS monitoring line in the Cloudera agent configuration file to bypass the check and restore agent communication:Open /etc/cloudera-scm-agent/config.ini using a text editor.Comment out the line monitored_nodev_filesystem_types=nfs,nfs4,tmpfs.Re

NiFi's content repository will hold on to claims until there are no active FlowFiles still anywhere on the canvas referencing that claim. This can result in very old or very large claims being left in the content repo using up space. Something as small as a single 0 byte FlowFile that sitting in some queue may end up preventing a multi-gigabyte content claim from being removed. The intent of this article is help users understand how to find those active FlowFiles and clear them form your dataflow.-There is no simple UI feature or NiFi rest-api endpoint users can use to return information linking FlowFiles to exiting content claims; however, all is not lost. Users can add an additional indexed field to your provenance configuration to so that is starts indexing the "ContentClaimIdentifier" associated with each FlowFile event generated. This would give you the ability to use Provenance to identify all the FlowFiles associated to specific claim in the content repo.-To add this, simply a