Community Articles

The NiFi community maintains a NiFi Docker image that comes very handy when you want to play with NiFi on your laptop. When you develop flows to run in a NiFi cluster with 2 or more nodes, though, there are things that you need to check and test differently from a single-node NiFi cluster. Recently, @Arqui shared a docker-compose configuration in this forum question that creates a 2-node NiFi cluster on Docker. The compose configuration still had a few issues to be ironed, though, and there were some manual steps you still had to perform before launching the cluster, like creating the TLS certificates for it. I found the idea interesting and spent some time working out the missing details. I also automated the TLS certificate creation and added a load-balancer to the compose configuration. Now, to use your own NiFi sanbox cluster, all you need to do is this: Download the docker-compose.yml file (see repo link below)Run "docker compose up -d"Connect to your

This guide provides a step-by-step approach to extracting data from SAP S/4HANA via OData APIs, processing it using Apache NiFi in Cloudera Data Platform (CDP), and storing it in an Iceberg-based Lakehouse for analytics and AI workloads. 1. Introduction1.1 Why Move SAP S/4HANA Data to a Lakehouse?SAP S/4HANA is a powerful ERP system designed for transactional processing, but it faces limitations when used for analytics, AI, and large-scale reporting:Performance Impact:Running complex analytical queries directly on SAP can degrade system performance.Limited Scalability:SAP systems are not optimized for big data workloads (e.g., petabyte-scale analytics).High Licensing Costs:Extracting and replicating SAP data for analytics can be expensive if done inefficiently.Lack of Flexibility:SAP’s data model is rigid, making it difficult to integrate with modern AI/ML tools.A Lakehouse architecture (built on Apache Ice

A resource in Cloudera Data Engineering is a named collection of files used by a job or a session. Resources can include application code, configuration files, custom Docker images, and Python virtual environment specifications (requirements.txt).CDE 1.24 introduces granular ACL's to prevent developers from utilizing and modifying resources in use by other users' Spark and Airflow jobs. In this article you will learn how to use the aforementioned ACL's with Jobs and Resources using the CDE CLI.Background Info on CDE and the CLICDE is the Cloudera Data Engineering Service, a containerized managed service for Cloudera Data Platform designed for Large Scale Batch and Streaming Pipelines with Spark, Airflow and Iceberg. It allows you to submit batch jobs to auto-scaling virtual clusters. As a Cloud-Native service, CDE enables you to spend more time on your applications, and less time on infrastructure.CDE allows you to create, manage, and schedule Apache Spark jobs without the overhead of

Cruise Control is an open-source solution to automate the dynamic workload rebalance and self-healing of large-scale Apache Kafka clusters. With the widespread adoption of Apache Kafka, many companies now maintain Kafka clusters with hundreds or even thousands of brokers. Operating large-scale clusters may lead to frequent broker failures and a substantial burden in redistributing workload. Cruise Control was designed to solve these operational scalability challenges.What is Cloudera Streams Messaging - Kubernetes Operator? Cloudera Streams Messaging - Kubernetes Operator brings enterprise-grade Kafka deployments to existing Kubernetes infrastructures. Cloudera Streams Messaging - Kubernetes Operator allows flexible, agile, and rapid deployment as well as scaling for variable workloads. This solution enables the deployment of Kafka and related components on existing, shared Kubernetes infrastructure, eliminating the need for dedicated infrastructureCruise Control is also part of t

What is Cloudera Global Support?Cloudera Global Support provides 24x7 customer assistance through a globally distributed team, combining deep technical expertise with proactive and predictive support capabilities to help customers successfully run Cloudera products.What issues are covered under Cloudera Support?Cloudera Support focuses on break/fix issues, including:Product crashes or failuresConfiguration issuesUnexpected product behaviorErrors during installation, upgrade, or deploymentBug fixes, patches, and workaroundsFor customer onboarding meant for new customers and/or new contacts within existing accounts to understand how Cloudera Support works, please click here.What is not covered under Cloudera Support ?Support does not include architecture design, performance tuning, training, custom solution development, or third-party system debugging. These requests are handled through Account Teams or Professional Services.How to access the support portal ?To access the support portal

With the addition of Python to Apache NiFi 2.0 and Cloudera Flow Management 4, developers now have a new way to rapidly develop processors and can leverage Python libraries for FlowFile processing, which is an exciting new feature. But what about NiFi’s scripting capabilities using Jython? In this post I will explore the history of Jython and Python in the Apache NiFi project, their capabilities, and the current state of Python and Jython in both Apache NiFi 2 and Cloudera Flow Management 4. NiFi’s Scripting Capabilities: Apache NiFi 0.5.0 introduced the ExecuteScript processor, allowing developers to interact with the NiFi SDK using several JVM-based scripting languages such as Groovy, Javascript, Lua, Clojure, and Jython, a JVM-based version of Python. The Jython language is compatible with Python,  so any  script you write in pure Python should work in ExecuteScript using Jython. “Pure” Python means the entire code is written in the Python language rather than ot

Spark Streaming Graceful shutdown 1. Introduction  In the first part of the blog post, we discussed the importance of graceful shutdown in Spark Streaming applications. We also saw different ways to shutdown a Spark Streaming application gracefully, including: Explicitly calling the JVM Shutdown Hook in the driver program. Using spark.streaming.stopGracefullyOnShutdown = true setting. In the second part, we will discuss a better way to gracefully shutdown the streaming application using other approaches.   2. Use an external approach to control internal program shutdown This approach allows an external process to control the shutdown of the internal program by sending a marker record, which is detected by the internal program. The internal program can then initiate a graceful shutdown procedure, ensuring that any unsaved data is saved and any resources are properly released before exiting. Benefits: It is more reliable than relying on the internal application to shut down

Introduction:In today's digital age, data security and compliance are paramount. Organizations handling sensitive information, especially in cloud environments, need robust tools to monitor and audit user activities. Cloudera offers a comprehensive auditing system to keep track of who's doing what in your environment. In this article, we'll delve into the world of audits in CDP Cloudera, explaining why they are essential and how to use them effectively.Why Audits Matter:Enhancing Security: Audits play a pivotal role in maintaining the security of your CDP Cloudera environment. By keeping a detailed log of user activities, audits help you identify suspicious or unauthorized actions promptly.Compliance Requirements: Many industries and organizations have stringent regulatory requirements. Audits help organizations adhere to these regulations by providing a clear record of data access and modifications.Investigation and Troubleshooting: When issues arise, audits serve as valuable tools fo

Objective In this article, you will create a CDE Spark Job with dependencies in multiple CDE Repositories. On CDE and the CLI CDE is the Cloudera Data Engineering Service, a containerized managed service for Cloudera Data Platform designed for large-scale batch and Streaming Pipelines with Spark, Airflow, and Iceberg. It allows you to submit batch jobs to auto-scaling virtual clusters. As a Cloud-Native service, CDE enables you to spend more time on your applications and less on infrastructure. CDE allows you to create, manage, and schedule Apache Spark jobs without the overhead of creating and maintaining Spark clusters. With CDE, you define virtual clusters with a range of CPU and memory resources, and the cluster scales up and down as needed to run your Spark workloads, helping to control your cloud costs. CDE provides a command-line interface (CLI) client. You can use the CLI to create and update jobs, view job details, manage job resources, run jobs, etc. Requirements The followin

Introduction Data Scientists are consistently looking to augment their data sources with additional datasets either to incorporate new features, refresh the latest data, or validate existing features. In Part 1 of this two-part article, we have covered how we can ingest data using Rest API endpoints.  In Part 2, we will explore how you can add data stored in an external file system into Cloudera Data Lake.  Specifically, we will ingest data from external S3 buckets of a publicly available dataset into a Cloudera Data Lake. Data Source  We will use the the publicly available Air Quality data made available by OpenAQ for this article. To know more about OpenAQ data available for developers , you can visit their website.  Pre-requisite  Access to the Cloudera AI platform  OpenAQ API Key ( available for free, on registration at their website www.openaq.org)  We plan to use a dataset from OpenAQ data source for getting Air Quality data

Introduction Data Scientists are consistently looking to augment their data sources with additional datasets either to incorporate new features, refresh the latest data, or validate existing features. In this 2 part article, we will explore two approaches to augment external data in the Cloudera Data Lake. We will explore how REST Catalog and external S3 buckets of a publicly available dataset can be accessed within ClouderaAI to ingest data.  This is  Part 1 of a two-part article. The Part 2 covers data ingestion from an external file system (s3). Data Source  In this article, we will use the publicly available Air Quality data made available by OpenAQ. Visit their website to learn more about OpenAQ data available for developers.  Pre-requisite  Access to the Cloudera AI platform  OpenAQ API Key ( available for free, on registration at their website www.openaq.org)  We plan to use a dataset from OpenAQ data source for getting Air Quality data for ci

Objective Deploying Dask Cuda Clusters on Kubernetes for distributed GPU workloads can require time, effort, and money, especially at Enterprise scale. You'd have to set up a Kubernetes cluster with GPU support, manage docker images, and dask cuda workers via complex deployment files. Cloudera AI simplifies distributed compute use cases in the context of Machine Learning and AI. In this article, you will learn how to quickly deploy a Dask Cuda Cluster in Cloudera AI using the cmlextensions library. Background Information Dask CUDA clusters enable scalable parallel computing on NVIDIA GPUs by leveraging Dask, a flexible parallel computing framework, in conjunction with the power of CUDA for GPU acceleration. These clusters allow users to distribute computation across multiple GPUs, significantly speeding up data processing and machine learning workflows, particularly for tasks involving large datasets or complex algorithms. Cloudera AI is a suite of artificial intelligence and machine l

Hive-Kafka Integration stepsThis procedure is well suited for a Kerberos and auto TLS-enabled cluster.To begin with please create client.properties files with entries as shown below in the bootstrap server(Active Controller Node check this from cm-->kafka--->instances-->xxxx.net.com(activecontroller):security.protocol=SASL_SSLsasl.mechanism=GSSAPIsasl.kerberos.service.name=kafkassl.truststore.location=/var/run/cloudera-scm-agent/process/71-kafka-KAFKA_CONNECT/cm-auto-global_truststore.jksssl.truststore.password=xxxxxxxssl.keystore.location=/var/run/cloudera-scm-agent/process/71-kafka-KAFKA_CONNECT/cm-auto-host_keystore.jksssl.keystore.password=xxxxxxxsasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true;2. Also it is important to export the Jaas.conf file from the respective path as given below.export KAFKA_OPTS="-Djava.security.auth.login.config=/tmp/jaas.conf"Topic creation in Kafka (please get a valid tkt if it is a Kerberos-enabled clus

In YARN, resource allocation discrepancies can occur due to the way resource calculation is handled. By default, resource availability is determined based on available memory.However, when CPU scheduling is enabled, resource calculation considers both available memory and vCores. As a result, in some scenarios, nodes may appear to allocate more vCores than the configured limit while simultaneously displaying lower available resources.This happens due to the way YARN dynamically assigns vCores based on workload demands rather than strictly adhering to preconfigured limits. Additionally, in cases where CPU scheduling is disabled, YARN relies solely on memory-based resource calculation. This may lead to negative values appearing in the YARN UI, which can be safely ignored, as they do not impact actual resource utilization.

Introduction This post explains the process of submitting Spark jobs on CDP Public Cloud Data Hub to the Livy server and provides a sample wrapper script for job submission. To understand the Livy Spark submit process, please refer to this post and understand how to arrive at the job configuration. The following steps submit the Spark Pi job from the spark_examples*.jar on S3. Steps Get the Livy endpoint for Data hub from the CDP control plane Ensure that you have the workload password set for the environment Download the Python wrapper script using the following command: wget https://raw.githubusercontent.com/karthikeyanvijay/cdp-publiccloud/master/datahub-scripts/livy-cdp-spark-submit/cdp_spark_submit.py Edit the script to modify the Livy endpoint, workload username, and password Copy the spark sample jar to the S3 hdfs dfs -cp /opt/cloudera/parcels/CDH-7.2.7-1.cdh7.2.7.p6.11615609/lib/spark/examples/jars/spark-examples_2.11-2.4.5.7.2.7.6-2.jar \ s3a://vkarthikeyan/ &

Cloudera on cloud offers a scalable, flexible, and cost-efficient solution for data management and analytics in the public cloud. It integrates data applications into a unified platform, enabling seamless data consolidation, advanced analytics, and machine learning. With its pay-as-you-go pricing model and managed services, Cloudera on cloud optimizes costs and simplifies operations. It enhances security and compliance, supports hybrid and multi-cloud environments, and provides tools for real-time data processing and self-service analytics. Cloudera on cloud empowers organizations to manage data efficiently while leveraging the latest cloud technologies.In comparison to Cloudera on premises, costs in public cloud can escalate quickly if not carefully monitored, a common challenge with all cloud workloads. However, with effective tracking and cost optimization strategies, public cloud adoptions can offer significant benefits in performance, scalability and cost savings over traditional

Pre-requisitesProcessStep 1: Install the [Impala ODBC Connector 2.7.0 for Cloudera Enterprise]Step 2: Copy the ODBC folder to Power BI Desktop folderStep 3: Edit the config fileStep 4: Run Power BI DesktopStep 5: Test Connection using ODBCStep 6: Fetch data in Power BI Desktop Pre-requisites Power BI Desktop Edition  Impala in CDP Private Cloud Base Impala ODBC Connector 2.7.0 for Cloudera Enterprise CDP Public Cloud Datahub + Kerberos + Power BI Desktop  (in the future) Process Step 1: Install the [Impala ODBC Connector 2.7.0 for Cloudera Enterprise]  Step 2: Copy the ODBC folder to Power BI Desktop folder Assume your Power BI Desktop is in [ C:\Program Files\Microsoft Power BI Desktop\ ], then please copy the ODBC Driver [ C:\Program Files\Cloudera ODBC Driver for Impala ] to [C:\Program Files\Microsoft Power BI Desktop\bin\ODBC Drivers\Cloudera ODBC Driver for Impala] Step 3: Edit the config file [ C:\Program Files\Microsoft Power BI Desktop

In this tutorial, we will learn how to create Apache Ozone volumes, buckets, and keys. After that, we will see how we can access Apache Ozone data in Apache Spark. Ozone Create the volume with the name vol1 in Apache Ozone.# ozone sh volume create /vol121/08/25 06:23:27 INFO rpc.RpcClient: Creating Volume: vol1, with root as owner. Create the bucket with the name bucket1 under vol1.# ozone sh bucket create /vol1/bucket121/08/25 06:24:09 INFO rpc.RpcClient: Creating Bucket: vol1/bucket1, with Versioning false and Storage Type set to DISK and Encryption set to false Create the employee.csv file to upload to Ozone.# vi /tmp/employee.csvid,name,age1,Ranga,332,Nishanth,43,Raja,60 Upload the employee.csv file to Ozone# ozone sh key put /vol1/bucket1/employee.csv /tmp/employee.csv Add the fs.o3fs.impl property to core-site.xml Go to Cloudera Manager > HDFS > Configuration > search for core-site.xml > Cluster-wide Advanced Configuration Snippet

Objective This project provides a quickstart for submitting Spark applications compiled from Scala code in Cloudera Data Engineering. It contains useful information targeted towards Spark developers and users who are migrating to Cloudera Data Engineering, such as: Context about Scala Jobs and the usage of Jars when running Spark Applications in CDE. Useful information about the CDE CLI and how it can simplify your migration efforts to CDE. Links to references and other recommended examples, such as submitting Spark App Code located in S3 and the CDE Spark Migration CLI Tool. Requirements To reproduce this example, you will need a CDE Service in Cloudera on Cloud or On Prem and a working installation of the CDE CLI. The Scala job application has already been compiled into a Jar via sbt and is available in the `target/scala-2.12/cde-scala-example_2.12-0.1.jar` path. Background Information on Scala, Spark and the Jar File Usage A JAR (Java ARchive) file is essentially a compressed file

ObjectiveIn this brief example you will learn how to use the CDE CLI to create a CDE Spark Job with PySpark and Scala app code located in an S3 bucket.RequirementsIn order to reproduce these examples in your CDE Virtual Cluster you will need:A Spark application in the form of a PySpark script or Scala jar, located in an S3 bucket.A working installation of the CDE CLI.A CDE 1.23 Service and a Spark 3 Virtual Cluster running in Cloudera on Cloud or Cloudera on Prem.Using Dependencies located in S3 with the Spark-SubmitCDE provides a command line interface (CLI) client. You can use the CLI to create and update jobs, view job details, manage job resources, run jobs, etc.Apache Spark Spark-Submit allows you to run a Spark job with application code located in an S3 bucket. The CDE CLI also provides this functionality.For example, in PySpark:  spark-submit \ --master yarn \ --deploy-mode cluster \ --py-files s3://your-bucket/path/to/dependency_one.zip, s3://your-bucket/path/to/de

Installing and Exploring Spark 2.0 with Jupyter Notebook and Anaconda Python in your laptop 1-Objective 2-Installing Anaconda Python 3-Checking Python Install 4-Installing Spark 5-Checking Spark Install 6-Launching Jupyter Notebook with PySpark 2.0.2 7-Exploring PySpark 2.0.2 a.Spark Session b.Read CSV i.Spark 2.0 and Spark 1.6 ii.Pandas c.Pandas DataFrames, Spark DataSets, DataFrames and RDDs d.Machine Learning Pipeline i.SciKit Learn ii.Spark MLLib, ML 8-Conclusion 1-Objective It is often useful to have python with the Jupyter notebook installed on your laptop in order to quickly develop and test some code ideas or to explore some data. Adding the ability to combine Apache Spark to this will also allow you to prototype ideas and exploratory data pipelines before hitting a Hadoop cluster and paying for Amazon Web Services. We leverage the power of the Python ecosystem with libraries such as Numpy (scientific computing library of high-level mathematical functions to operate

While generative AI dominates today's headlines, traditional predictive machine learning models continue to drive critical business decisions across industries. To ensure predictive models achieve a solid ROI, well after models are initially deployed,  establishing a Machine Learning Operations (MLOps) plan is essential. MLOps is the practice of streamlining the entire lifecycle of machine learning models—from development and training to deployment, monitoring, and maintenance—in a repeatable, scalable, and governable way. Think of it as bringing software engineering discipline to machine learning, ensuring that your AI investments don't remain theoretical exercises but become dependable business assets that continue to deliver value over time. Without robust MLOps practices, models often degrade in production as data shifts over time. What begins as an impressive prototype can quickly become unreliable, leading to poor decision quality with real financial consequences. Poor model

OverviewTPC-DS (Transaction Processing Performance Council - Decision Support) is a benchmark suite for evaluating the performance of data warehouse and decision support workloads. This guide provides a step-by-step approach to setting up and executing TPC-DS on Cloudera Datahub Spark Cluster, covering data generation, query execution, and performance analysis.Note: Required files for implementation can be found in the attached GitHub repository:👉tpcds-cloudera GitHub RepositoryDatagenThis document provides step-by-step instructions to set up and run TPC-DS Data Generation on a Cloudera Datahub Spark Cluster.Step 1: Create spark-sql-perf DirectoryCreate the directory on the master node:The directory structure and name are not mandatory, but ensure to update the commands accordingly if changed.Step 2: Setup Storage Bucket DirectoryCreate the TPC-DS directory inside the data bucket:Step 3: Copy the Datagen JAR to Master NodeCopy the spark-tpcds-datagen-2.12.jar from the local system to

CREATING A CDE REPOSITORY WITH A PRIVATE GIT REPOSITORY OBJECTIVE In this article, you will learn how to create a CDE Repository by cloning a private Git repository using CDE Credentials. Cloudera Data Engineering is a cloud-based service that helps businesses build, manage, and scale data pipelines. It is based on Apache Spark and integrates with other Cloudera services. Git repositories allow teams to collaborate, manage project artifacts, and promote applications from lower to higher environments. CDE Repositories can be used to clone git repositories into a Virtual Cluster to make the imported files available to CDE Jobs as reusable dependencies. CDE currently supports Git providers such as GitHub, GitLab, and Bitbucket. CDE Credentials allow you to store secrets such as usernames, passwords, and tokens, in secure, reusable objects associated with CDE Jobs, Custom Runtimes and Repositories. REQUIREMENTS This example was built with CDE 1.23. To reproduce the commands in your environ

OBJECTIVE:Step-by step instructions for enabling the git persistence provider for NiFi in Cloudera Flow Management (CFM) on Cloudera Manager. OVERVIEW:The default install of early releases of Cloudera Flow Management (CFM) includes only basic configurations for NiFi Registry in Cloudera Manager.This article explains how to reconfigure the Nifi Registry persistence providers to use git instead of the filesystem provider. PREREQUISITES:CFM (CDF) 1.0.1 or later and CM 5.16 or CM 6.2 or later ADDING NiFi METRICS TO CM:1. Pick a location where the Nifi Registry will store local git repository files. This location has to be accessible to the NiFi Registry service account (typically: nifiregistry). It must also be initialized as a git repository using git init and linked to a remote repository using git remote add origin. (Alternatively, an existing git repository can be cloned to create the folder using git clone). The substitution variable ${nifi.registry.working.directory} c