Resources

Pre-requisitesProcessStep 1: Install the [Impala ODBC Connector 2.7.0 for Cloudera Enterprise]Step 2: Copy the ODBC folder to Power BI Desktop folderStep 3: Edit the config fileStep 4: Run Power BI DesktopStep 5: Test Connection using ODBCStep 6: Fetch data in Power BI Desktop Pre-requisites Power BI Desktop Edition  Impala in CDP Private Cloud Base Impala ODBC Connector 2.7.0 for Cloudera Enterprise CDP Public Cloud Datahub + Kerberos + Power BI Desktop  (in the future) Process Step 1: Install the [Impala ODBC Connector 2.7.0 for Cloudera Enterprise]  Step 2: Copy the ODBC folder to Power BI Desktop folder Assume your Power BI Desktop is in [ C:\Program Files\Microsoft Power BI Desktop\ ], then please copy the ODBC Driver [ C:\Program Files\Cloudera ODBC Driver for Impala ] to [C:\Program Files\Microsoft Power BI Desktop\bin\ODBC Drivers\Cloudera ODBC Driver for Impala] Step 3: Edit the config file [ C:\Program Files\Microsoft Power BI Desktop

In this tutorial, we will learn how to create Apache Ozone volumes, buckets, and keys. After that, we will see how we can access Apache Ozone data in Apache Spark. Ozone Create the volume with the name vol1 in Apache Ozone.# ozone sh volume create /vol121/08/25 06:23:27 INFO rpc.RpcClient: Creating Volume: vol1, with root as owner. Create the bucket with the name bucket1 under vol1.# ozone sh bucket create /vol1/bucket121/08/25 06:24:09 INFO rpc.RpcClient: Creating Bucket: vol1/bucket1, with Versioning false and Storage Type set to DISK and Encryption set to false Create the employee.csv file to upload to Ozone.# vi /tmp/employee.csvid,name,age1,Ranga,332,Nishanth,43,Raja,60 Upload the employee.csv file to Ozone# ozone sh key put /vol1/bucket1/employee.csv /tmp/employee.csv Add the fs.o3fs.impl property to core-site.xml Go to Cloudera Manager > HDFS > Configuration > search for core-site.xml > Cluster-wide Advanced Configuration Snippet

Objective This project provides a quickstart for submitting Spark applications compiled from Scala code in Cloudera Data Engineering. It contains useful information targeted towards Spark developers and users who are migrating to Cloudera Data Engineering, such as: Context about Scala Jobs and the usage of Jars when running Spark Applications in CDE. Useful information about the CDE CLI and how it can simplify your migration efforts to CDE. Links to references and other recommended examples, such as submitting Spark App Code located in S3 and the CDE Spark Migration CLI Tool. Requirements To reproduce this example, you will need a CDE Service in Cloudera on Cloud or On Prem and a working installation of the CDE CLI. The Scala job application has already been compiled into a Jar via sbt and is available in the `target/scala-2.12/cde-scala-example_2.12-0.1.jar` path. Background Information on Scala, Spark and the Jar File Usage A JAR (Java ARchive) file is essentially a compressed file

ObjectiveIn this brief example you will learn how to use the CDE CLI to create a CDE Spark Job with PySpark and Scala app code located in an S3 bucket.RequirementsIn order to reproduce these examples in your CDE Virtual Cluster you will need:A Spark application in the form of a PySpark script or Scala jar, located in an S3 bucket.A working installation of the CDE CLI.A CDE 1.23 Service and a Spark 3 Virtual Cluster running in Cloudera on Cloud or Cloudera on Prem.Using Dependencies located in S3 with the Spark-SubmitCDE provides a command line interface (CLI) client. You can use the CLI to create and update jobs, view job details, manage job resources, run jobs, etc.Apache Spark Spark-Submit allows you to run a Spark job with application code located in an S3 bucket. The CDE CLI also provides this functionality.For example, in PySpark:  spark-submit \ --master yarn \ --deploy-mode cluster \ --py-files s3://your-bucket/path/to/dependency_one.zip, s3://your-bucket/path/to/de

Installing and Exploring Spark 2.0 with Jupyter Notebook and Anaconda Python in your laptop 1-Objective 2-Installing Anaconda Python 3-Checking Python Install 4-Installing Spark 5-Checking Spark Install 6-Launching Jupyter Notebook with PySpark 2.0.2 7-Exploring PySpark 2.0.2 a.Spark Session b.Read CSV i.Spark 2.0 and Spark 1.6 ii.Pandas c.Pandas DataFrames, Spark DataSets, DataFrames and RDDs d.Machine Learning Pipeline i.SciKit Learn ii.Spark MLLib, ML 8-Conclusion 1-Objective It is often useful to have python with the Jupyter notebook installed on your laptop in order to quickly develop and test some code ideas or to explore some data. Adding the ability to combine Apache Spark to this will also allow you to prototype ideas and exploratory data pipelines before hitting a Hadoop cluster and paying for Amazon Web Services. We leverage the power of the Python ecosystem with libraries such as Numpy (scientific computing library of high-level mathematical functions to operate

While generative AI dominates today's headlines, traditional predictive machine learning models continue to drive critical business decisions across industries. To ensure predictive models achieve a solid ROI, well after models are initially deployed,  establishing a Machine Learning Operations (MLOps) plan is essential. MLOps is the practice of streamlining the entire lifecycle of machine learning models—from development and training to deployment, monitoring, and maintenance—in a repeatable, scalable, and governable way. Think of it as bringing software engineering discipline to machine learning, ensuring that your AI investments don't remain theoretical exercises but become dependable business assets that continue to deliver value over time. Without robust MLOps practices, models often degrade in production as data shifts over time. What begins as an impressive prototype can quickly become unreliable, leading to poor decision quality with real financial consequences. Poor model

OverviewTPC-DS (Transaction Processing Performance Council - Decision Support) is a benchmark suite for evaluating the performance of data warehouse and decision support workloads. This guide provides a step-by-step approach to setting up and executing TPC-DS on Cloudera Datahub Spark Cluster, covering data generation, query execution, and performance analysis.Note: Required files for implementation can be found in the attached GitHub repository:👉tpcds-cloudera GitHub RepositoryDatagenThis document provides step-by-step instructions to set up and run TPC-DS Data Generation on a Cloudera Datahub Spark Cluster.Step 1: Create spark-sql-perf DirectoryCreate the directory on the master node:The directory structure and name are not mandatory, but ensure to update the commands accordingly if changed.Step 2: Setup Storage Bucket DirectoryCreate the TPC-DS directory inside the data bucket:Step 3: Copy the Datagen JAR to Master NodeCopy the spark-tpcds-datagen-2.12.jar from the local system to

CREATING A CDE REPOSITORY WITH A PRIVATE GIT REPOSITORY OBJECTIVE In this article, you will learn how to create a CDE Repository by cloning a private Git repository using CDE Credentials. Cloudera Data Engineering is a cloud-based service that helps businesses build, manage, and scale data pipelines. It is based on Apache Spark and integrates with other Cloudera services. Git repositories allow teams to collaborate, manage project artifacts, and promote applications from lower to higher environments. CDE Repositories can be used to clone git repositories into a Virtual Cluster to make the imported files available to CDE Jobs as reusable dependencies. CDE currently supports Git providers such as GitHub, GitLab, and Bitbucket. CDE Credentials allow you to store secrets such as usernames, passwords, and tokens, in secure, reusable objects associated with CDE Jobs, Custom Runtimes and Repositories. REQUIREMENTS This example was built with CDE 1.23. To reproduce the commands in your environ

OBJECTIVE:Step-by step instructions for enabling the git persistence provider for NiFi in Cloudera Flow Management (CFM) on Cloudera Manager. OVERVIEW:The default install of early releases of Cloudera Flow Management (CFM) includes only basic configurations for NiFi Registry in Cloudera Manager.This article explains how to reconfigure the Nifi Registry persistence providers to use git instead of the filesystem provider. PREREQUISITES:CFM (CDF) 1.0.1 or later and CM 5.16 or CM 6.2 or later ADDING NiFi METRICS TO CM:1. Pick a location where the Nifi Registry will store local git repository files. This location has to be accessible to the NiFi Registry service account (typically: nifiregistry). It must also be initialized as a git repository using git init and linked to a remote repository using git remote add origin. (Alternatively, an existing git repository can be cloned to create the folder using git clone). The substitution variable ${nifi.registry.working.directory} c

 SummaryThis article details the resolution of a startup failure for Knox Gateway roles during a Data Hub upgrade, involving the reconfiguration of a missing knox_pac4j_password.SymptomsWhile upgrading the Data Hub cluster, Knox Gateway services failed to start.Error message: java.lang.IllegalArgumentException: The variable [${knox_pac4j_password}] does not have a corresponding value.The upgrade process was halted due to this failure.CauseThe probable cause of the startup failure was an improperly generated or missing knox_pac4j_password during the upgrade process.The built-in upgrade handler should have automatically generated the Knox secret, but it did not.InstructionsTo resolve the issue, the following steps were taken:Verify if an entry for knox_pac4j_password exists in the Cloudera Manager database.Execute SQL query: SELECT * FROM configs WHERE attr LIKE '%knox_pac4j_password%';Generate a 16-bit UUID random password.Use OpenSSL: openssl ran

SummaryResolved service errors on Datanode#09 by identifying and addressing NFS mount issues. The Cloudera agent was reconfigured to bypass NFS checks temporarily, allowing services to return to a healthy state.SymptomsMultiple service errors reported on Datanode#09.Cloudera agent was not in contact with Cloudera Manager.Filesystem usage for certain nodev filesystems was unknown due to an inactive worker process.CauseThe NFS partition was not properly mounted on the host, which led to problems with the Cloudera agent's health checks.The Cloudera agent configuration was set to monitor NFS mounts (monitored_nodev_filesystem_types=nfs,nfs4,tmpfs), which failed due to the unmounted NFS partition.InstructionsTemporarily comment out the NFS monitoring line in the Cloudera agent configuration file to bypass the check and restore agent communication:Open /etc/cloudera-scm-agent/config.ini using a text editor.Comment out the line monitored_nodev_filesystem_types=nfs,nfs4,tmpfs.Re

NiFi's content repository will hold on to claims until there are no active FlowFiles still anywhere on the canvas referencing that claim. This can result in very old or very large claims being left in the content repo using up space. Something as small as a single 0 byte FlowFile that sitting in some queue may end up preventing a multi-gigabyte content claim from being removed. The intent of this article is help users understand how to find those active FlowFiles and clear them form your dataflow.-There is no simple UI feature or NiFi rest-api endpoint users can use to return information linking FlowFiles to exiting content claims; however, all is not lost. Users can add an additional indexed field to your provenance configuration to so that is starts indexing the "ContentClaimIdentifier" associated with each FlowFile event generated. This would give you the ability to use Provenance to identify all the FlowFiles associated to specific claim in the content repo.-To add this, simply a

Servers used in this document:node1.localdomain = HAProxy servernode2.localdomain = NameNode1node3.localdomain = NameNode21. Install HAProxy[root@node1 ~]# yum install -y haproxy2. Set up HAProxy with minimum configuration[root@node1 ~]# cp -p /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.orig[root@node1 ~]# vim /etc/haproxy/haproxy.cfg... (snip) ...#---------------------------------------------------------------------# main frontend which proxys to the backends#---------------------------------------------------------------------frontend main *:50070 default_backend app#---------------------------------------------------------------------# round robin balancing between the various backends#---------------------------------------------------------------------backend app balance roundrobin option httpchk GET /webhdfs/v1/?op=CHECKACCESS http-check expect rstatus ([23][0-9][0-9]|401) server node2 node2.localdomain:50070 check server node3 node3.localdomain:50070 chec

It's early 2025 and agentic workflows are the hottest commodities within AI. Giving large language models (LLMs) the ability to use tools to complete actions opens up a huge range of use cases. At the same time, it's important to ensure that the AI agents we build are secure.Why AI Agents?AI Agents have the potential to transform society and how we leverage technology to complete routine tasks. Some key use cases for AI Agents can involve:Automation Customer Service TasksOrchestrating Business Processes & WorkflowsTaking Actions Based on Unstructured DataIn this article you will get familiar the CrewAI library and how we can build fully private AI agents with CrewAI and Cloudera's AI Inference Service. Powered by Nvidia NeMo Inference Microservices (NIM), Cloudera's AI Inference Service gives you the ability to deploy private LLMs within your infrastructure. This article is designed to walk you through the setup and application of these tools in an agentic worfklow context to creat

Prerequisites: This article assumes some knowledge of Spark and Iceberg. These examples make use of Cloudera Data Engineering and Cloudera Data Warehouse.Note: Iceberg branch-based WAP requires Spark 3.5 or above. Motivation (simulated scenario) Let's imagine we have a high-traffic table many users use in the platform. Each day, before the start of a work day, a Spark job populates this table with new records. On most days, this simple architecture will work quite well. However, this morning the Spark job had a data error and wrote out a batch of duplicate records. This data quality issue was reported by one of the users and the data engineers are now trying to fix it. However, during this time, access to the high-traffic table is obviously disrupted. Of course, data quality checks can be implemented to catch issues like this earlier. However, this still leaves you little time to roll back changes or fix the data issues before users are affected. Write Audit Publish a pattern These t

JupyterLab is a powerful, web-based interactive development environment widely used for data science, machine learning, and Spark application development. It extends the classic Jupyter Notebook interface, offering a more flexible and integrated workspace.For Spark application development, JupyterLab provides the following advantages:1. Interactive Exploration: Run Spark queries and visualize results interactively, making it ideal for data exploration and debugging.2. Rich Visualization: Seamlessly integrate with Python visualization libraries like Matplotlib, Seaborn, and Plotly to analyze and interpret Spark data.3. Ease of Integration: Use PySpark or Sparkmagic to connect JupyterLab with Spark clusters, enabling distributed data processing from the notebook environment.Spark Connect is a feature introduced in Apache Spark that provides a standardized, client-server architecture for connecting to Spark clusters. It decouples the client from the Spark runtime, allowing users to intera

PyCharm is a popular integrated development environment (IDE) for Python, developed by JetBrains. It provides a comprehensive set of tools designed to boost productivity and simplify the coding process. When working with Apache Spark, PyCharm can be an excellent choice for managing Python-based Spark applications. Its features include:Code Editing and Debugging: PyCharm offers intelligent code completion, syntax highlighting, and robust debugging tools that simplify Spark application development.Virtual Environments and Dependency Management: PyCharm makes it easy to configure Python environments with Spark libraries and manage dependencies.Notebook Support: With built-in support for Jupyter Notebooks, PyCharm allows you to work interactively with data, making it easier to visualize and debug Spark pipelines.Version Control: PyCharm integrates with Git and other version control systems, simplifying collaboration and project management.Spark Connect is a feature introduced in Apach

Spark GraphFrames is a package within Apache Spark that allows users to perform graph processing operations on data using a DataFrame-based approach. It will enable you to perform various graph operations like finding connected components, calculating shortest paths, identifying triangles, and more. Real world applications include social network analysis, recommendation systems, web link structures, flight connections, and other scenarios where relationships between data points are crucial. Cloudera AI (CAI) is a cloud-native service within the Cloudera that enables enterprise data science teams to collaborate across the full data lifecycle. It provides immediate access to enterprise data pipelines, scalable compute resources, and preferred tools, streamlining the process of moving analytic workloads from research to production. Using Spark GraphFrames in Cloudera AI requires minimum configuration effort. This quickstart provides a basic example so you can get started quickly. Requirem

To Fix under-replicated blocks in HDFS, below is quick instruction to use:####Fix under-replicated blocks### su - <$hdfs_user>bash-4.1$ hdfs fsck / | grep 'Under replicated' | awk -F':' '{print $1}' >> /tmp/under_replicated_files -bash-4.1$ for hdfsfile in `cat /tmp/under_replicated_files`; do echo "Fixing $hdfsfile :" ; hadoop fs -setrep 3 $hdfsfile; done

Problem Statement: Model governanceData science and model building are prevalent activities that bring new and innovative value to enterprises. The more prevalent this activity becomes, the more problematic model governance becomes. Model governance typically centers on these questions: What models were deployed? when? where? What was the serialized object deployed?Was deployment to a microservice, a Spark context, other?What version was deployed?How do I trace the deployed model to its concrete details: the code, its training data, owner, Read.me overview, etc?Apache Atlas is the central tool in organizing, searching and accessing metadata of data assets and processes on your Hadoop platform. Its Rest API can push metadata from anywhere, so Atlas can also represent metadata off your Hadoop cluster.Atlas lets you define your own types of objects and inherit from existing out-of-the box types. This lets you store whatever metadata you want to store, and to tie this into Atlas's pow

Why this article is created?The main reason for creating and listing all the legacy configurations for Spark 3 is to ease the migration from Spark 2 to Spark 3.Main concern here was that during migration Spark 3 behaves different than Spark 2 due to upstream changes and causing the same code to work differently.Legacy configuration in Spark 3Property nameDefaultMeaningspark.sql.catalog.your_catalog_name.pushDownAggregatespark.sql.catalog.your_catalog_name.pushDownLimitspark.sql.catalog.your_catalog_name.pushDownOffsetTrueSince Spark 3.5, the JDBC options related to DS V2 pushdown are true by default. These options include: pushDownAggregate, pushDownLimit, pushDownOffset and pushDownTableSample.To restore the legacy behavior, please set them to false. e.g. set spark.sql.catalog.your_catalog_name.pushDownAggregate to false.spark.sql.legacy.negativeIndexInArrayInsertFalseSince Spark 3.5, the array_insert function is 1-based for negative indexes. It inserts new element at the end of input

Spark Python Integration Test Result Exceptions   In this article, just I talk about exceptions and their Python and Spark versions. Keep on watching this article where I will add some more exceptions and solutions. 1. TypeError: an integer is required (got type bytes) Exception: Traceback (most recent call last): File "/opt/pyspark_udf_example.py", line 3, in <module> from pyspark.sql import SparkSession File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 655, in _load_unlocked File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible File "<frozen zipimport>", line 259, in load_module File "/opt/spark/python/lib/pyspark.zip/pyspark/__init__.py", line 46, in <module> File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._b

IntroductionImpala is a horizontally scalable, distributed database engine that is best suited for fast BI workloads. Apache Iceberg is an open table format that has brought new and exciting capabilities to the big data world with support for deletes, updates, time travel, partition management, and more. With Iceberg, users can insert data into tables that have advanced and evolving partition layouts. Data warehouse users often need to comply with regulations like GDPR and CCPA, requiring them to delete records from their warehouses. They might also need to update their data from time to time. These operations often require regular removal of old partitions. Impala and Iceberg can easily handle these use cases. Frequent write operations can CREATE TABLE ice_tbl (i INT, s STRING, ts TIMESTAMP)STORED BY ICEBERG; potentially lead to performance issues, including the small file problem, or the overhead of handling lots of delete files. To resolve these issues, Impala can als

For users experiencing trouble uploading a file to a case using the standard ‘UPLOAD FILE(s)’ method from the Case Attachment tab, the following alternatives exist:Upload files from the command line30GB LimitThis method utilizes a java program to allow case attachments to be uploaded via the command line. This method is intended for those who are experiencing trouble with the case attachment UI or who prefer to upload a file from the command line.  This method also supports proxy parametersHow to use:Download the JARRun the program:Usage: java -jar cloudera.attachment.cli.jar [case_number] [email_address] [file_name...]Example:  java -jar cloudera.attachment.cli.jar 55555 customer@company.com logs1.log logs2.logThe above example uploads logs1.log and logs2.log to case number 55555 from customer@company.comSupported proxy command line properties include:  -Dhttp.proxyHost=, -Dhttp.proxyPort=, -Dhttp.proxyUsername=, -Dhttp.proxyPassword=Example: java -j

IntroductionLet's be clear up front: by "tactical", I mean, "not recommended for production use" 🙂 There are several better ways to achieve code re-use in Cloudera Data Engineering, including adding custom operators and libraries at the Virtual Cluster level, and @VishRajagopalan's CDE Dynamic Package Loader.However, if you just need something more "quick and dirty", then read on to learn how to load Python code from Resources in Cloudera Data Engineering.To quote the official documentation:A Resource in Cloudera Data Engineering is a named collection of files used by a job or a session. Resources can include application code, configuration files, custom Docker images, and Python virtual environment specifications (requirements.txt).The key to understanding how to leverage Resources to provide custom code to your Airflow Jobs in Cloudera Data Engineering is to know how exactly Resources are made available at run-time. Once you know that, the rest is just "normal" Python