Resources

In the GenAI era, I’ve really enjoyed how tools like LLMs can boost productivity in day-to-day operations. However, I found that NiFi’s Web UI doesn’t integrate very naturally with GenAI workflows. I also explored options like MCP Server to bridge that gap, but it requires additional setup and a specific skillset.Recently, I discovered a much simpler approach—leveraging NiFi’s REST APIs directly. Without any MCP setup, we can use LLMs as an operations co-pilot to extract, audit, and document NiFi flows and Ranger policies.What Is Vibe Coding for NiFi Development?Vibe coding means you describe your intent in natural language and let AI write the code. Applied to CDP NiFi flow development, it looks like this:You say: "Show me all Process Groups and their processor counts" AI does: Generates the correct curl commands, calls the NiFi REST API, parses the JSON, and returns a clean summary tableAs a NiFi flow developer, you spend most of your time designing data pipelines -- not memorizing A

Running Apache Airflow on Kubernetes brings incredible scalability, but when DAGs stall, tasks disappear, or the scheduler goes silent, navigating the infrastructure layer can be daunting. This guide breaks down essential debugging commands into six critical pillars, explaining why you need them, when to use them, and what to look out for during an incident. 1. Kubernetes Pods, Nodes, and Events Before digging into Airflow application logs, check the health of your underlying Kubernetes infrastructure. Pod evictions, scheduling failures, or resource constraints are often the root cause of "mysterious" task failures. List pods and states kubectl get pods -n <namespace> -o wide When to use: Your first line of defense when tasks aren't picking up or the UI is unresponsive. What to look for: Look for pods in CrashLoopBackOff, OOMKilled, or Pending states. The -o wide flag gives you the IP and node name, helping you identify if a specific Kubernetes node is failing. Iden

AI has moved from boardroom ambition to competitive necessity. Enterprises that can deploy production-grade AI on their most sensitive data, customer records, financial transactions, and patient histories will outpace those that cannot. Yet the majority of AI initiatives stall not because of a lack of talent or technology, but because of one intractable problem: how do you feed sensitive data into AI systems without violating privacy regulations, triggering compliance reviews, or opening the door to data breaches? The answer, until now, has been: you don't. You either expose data and accept the risk, or you exclude it and accept the limitations. Neither option is acceptable for a business serious about AI. Three Roadblocks Killing AI Initiatives Across industries, the same three barriers appear again and again when enterprises try to operationalize AI on sensitive data: 1. The AI-Ready Data Gap Legacy systems weren’t built for AI-scale speed and volume. You’re forced to choose: exposes

Spark Python Supportability MatrixThe Spark Python Supportability Matrix serves as an essential tool for determining which Python versions are compatible with specific Spark versions. This matrix provides a detailed overview of the compatibility levels for various Python versions across different Spark releases.Spark VersionPython Min Supported VersionPython Max Supported VersionPythonv 2.7Pythonv3.4Pythonv3.5Pythonv 3.6Pythonv3.7Pythonv3.8Pythonv3.9Pythonv3.10Pythonv3.113.5.53.83.11NoNoNoNoNoYesYesYesYes3.5.43.83.11NoNoNoNoNoYesYesYesYes3.5.33.83.11NoNoNoNoNoYesYesYesYes3.5.23.83.11NoNoNoNoNoYesYesYesYes3.5.13.83.11NoNoNoNoNoYesYesYesYes3.5.03.83.11NoNoNoNoNoYesYesYesYes3.4.23.73.11NoNoNoNoYesYesYesYesYes3.4.13.73.11NoNoNoNoYesYesYesYesYes3.4.03.73.11NoNoNoNoYesYesYesYesYes3.3.33.73.10NoNoNoNoYesYesYesYesNo3.3.23.73.10NoNoNoNoYesYesYesYesNo3.3.13.73.10NoNoNoNoYesYesYesYesNo3.3.03.73.10NoNoNoNoYesYesYesYesNo3.2.43.63.9NoNoNoYesYesYesYesNoNo3.2.33.63.9NoNoNoYesYesYesYesNoNo3.2.23.63.9No

PrerequisitesBefore beginning the key rotation process, ensure the following are in place:Virtual machine account access or the ability to create them via ssh-keygen: You must have the necessary permissions to manage Virtual Machines and Key Vaults in your cloud provider environment or via bash command line interface.New SSH key pair: Generate a new, secure SSH key pair (public and private).AssumptionsYou have an old key called id_rsa and id_rsa.pub.You have generated a new key pair, either via your cloud provider SSH Keys service or via ssh-keygen via Bash CLI. We will assume; for the purpose of this guide, that the new keys are id_rsa_new and id_rsa_new.pub.You have access to the Cloudera Control Plane. You have PowerUser or similar permissions to update the Summary tab values in your environment.1. Generate a New SSH Key PairA new, unique key pair must be generated for the rotation.Key StepCommand (Linux/macOS)DescriptionGenerate keyssh-keygen -t rsa -b 4096 -f ~/.ssh

We are all using AI to write code, but when it comes to Apache NiFi, the current landscape often resembles the Wild West. Whether you are generating synthetic data scripts or translating complex machine learning models, Large Language Models (LLMs) are incredible accelerators. However, if you ask an AI to write a native Apache NiFi 2.0 Python processor from scratch, there is a very high probability it will confidently hand you code that instantly breaks your canvas.NiFi 2.0’s Python API is relatively new, and most AI training data is heavily saturated with legacy NiFi 1.x ExecuteScript solutions (using Jython or Groovy). Even when an AI correctly identifies the 2.0 API, it frequently misconfigures the underlying Java-to-Python bridge, resulting in “ghost” processors with dashed lines and missing routing relationships.In this post, I am going to share the exact methodology I used to leverage AI for writing custom NiFi processors safely, ensuring my dataflows operate seamlessly

Optimizing Hive queries is crucial for achieving better performance and scalability in a data warehouse environment. Here are some tips and best practices for optimizing Hive queries: Partitioning: Partitioning your data can significantly improve query performance by reducing the amount of data scanned during query execution. Partition your tables based on commonly filtered columns, such as date or category. Use static partitioning for columns with a limited number of distinct values and dynamic partitioning for columns with high cardinality. Consider using partitioned tables for time-series data to improve query performance for date-range queries. Bucketing: Bucketing distributes data into a fixed number of buckets based on the hash value of one or more columns. Use bucketing to distribute data across files and improve data locality evenly. Choose the number of buckets wisely based on the size of your data and the available resources. Bucketing is particularly useful for optimi

IntroductionApache NiFi 1.0 was recently released and being integrated into Hortonworks Data Flow (HDF) that will be release very soon.In this easy tutorial we will see how we can stream data from CSV format into Hive tables directly and start working on it right away without a single line of coding to set up the streaming. Pre-requisitesIn order to run this tutorial successfully you need to download the Following:NiFi 1.0 or higher, you can download it from hereHDP Sandbox 2.4 or higher, you can download it from hereDownload the Olympics CSV data from the attachment list below. Changing NiFi Port (Optional)Since Ambari and NiFi both uses port 8080, you will have problems starting NiFi if you are running the sandbox and NiFi on the same machine.Once NiFi is downloaded, uncompress it and go to /nifi/conf/nifi.properties and change the port no to 8089 as follows:nifi.web.http.port=8089 Starting NiFi and the SandboxOnce NiFi is downloaded, uncompress and start it using the command:/nifi/

Setting up a new MacBook for Data Engineering is quite a chore. In this guide, we will take you from a fresh macOS installation to running the full suite of Cloudera Streaming Operators (CFM, CSA, CSM) on a local Minikube cluster.   Please note that we are utilizing the Evaluation examples for these services. This is a critical step for any engineer; mastering the foundational configurations is essential before moving into complex, secure, and high-availability production architectures. Let’s get started! My MacBook Setup First, we need the “Swiss Army Knife” of macOS: homebrew. We will use it to install our core CLI tools, container runtimes, and local Kubernetes environment. # Install Homebrew (if not already installed) /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" # Install Core Tools brew install git docker python minikube kubernetes-cli k9s helm # Start Docker Desktop (or Colima) open

Here is the steps to access Ambari UI through Knox. This is tried and tested with Ambari 2.4 and HDP2.51. Make sure Knox is configured properly and it works fine.2. ssh to Knox gateway host and go to /var/lib/knox/data-2.5.3.0-37/services3. download the configurations from https://github.com/apache/knox/tree/v0.11.0/gateway-service-definitions/src/main/resources/services/ambariui/2.2.0/ URL.4. make sure your folder structure should look alike /var/lib/knox/data-2.5.3.0-37/services/ambariui/2.2.0 and should have rewrite.xml and service.xml files5. change the owner/Group permissions to Knox for /var/lib/knox/data-2.5.3.0-37/services/ambariui/ and subdirectory6. Go to Knox configurations Modify "Advanced topology" with below service tag <service> <role>AMBARIUI</role> <url>http://AMBARIHOST:8080</url> </service>7. Restart Knox service.8. You should be able to access Ambari-server UI from the below URLhtt

Hi. I need to specify multiple recipients for the emails we receive when the support cases related to our account are updated e.g. from Proactive Support. Can I manage our account myself online? Thanks,Ian.  

SummaryIf you have lost your RMON db password and need to retrieve it, in CM GUI it is **** out.  This article will tell you have you can retrieve it in plain text. SymptomsAny customer needing to get into RMON db but not having the password would need to get the password to get in (assuming they do not have DB ROOT)InstructionsTo retrieve the password we can do an API call that gives us the RMON db config info. api/v31/cm/service/roleConfigGroups/mgmt-REPORTSMANAGER-BASE/config an example would be:curl -u admin:admin GET http://ccycloud-1.oteixeira-ubuntu2004.root.comops.site:7180/api/v31/cm/service/roleConfigGroups/mgmt-REPORTSMANAGER-BASE/config​And the results would appear similar to:{ "items" : [ { "name" : "headlamp_database_host", "value" : "ccycloud-1.oteixeira-ubuntu2004.root.comops.site:3306" }, { "name" : "headlamp_database_name", "value" : "reportsmanager9f05b8e6450d2994f37859d1ee3f1967" }, { "name" : "headlamp_database_password", "v

Summary Yarn distributed shell test job fails when running with Ldap group mapping turned on with basic CDP 7.1.9 SP1 cluster using Java 17 JDK.24/12/11 15:21:29 ERROR distributedshell.ApplicationMaster: Error running ApplicationMasterjava.lang.IllegalAccessError: class org.apache.hadoop.security.LdapGroupsMapping (in unnamed module @0x3901d134) cannot access class com.sun.jndi.ldap.LdapCtxFactory (in module java.naming) because module java.naming does not export com.sun.jndi.ldap to unnamed module @0x3901d134at org.apache.hadoop.security.LdapGroupsMapping.<clinit>(LdapGroupsMapping.java:264)Symptoms Applies to  New clusters using CDP 7.1.9 SP1 and Java 17 JDK with Ldap group mapping on.  InstructionsNeed to add In Yarn - Configuration - search for yarn.nodemanager.admin-env and change the NodeManager Group (all of them if you have more then one) to: JDK_JAVA_OPTIONS=--add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.util=ALL-

Apache NiFi 2.X or CFM Version 4.X now provide an ability to use Bitbucket as flow version control in place of NiFi Registry. This new Registry Client is built with the name BitbucketFlowRegistryClient to connect to Bitbucket To set up BitbucketFlowRegistryClient following information is required from the Bitbucket account as a prerequisite; if not present must be created  Workspace Name Repository Name Authentication Type  Default Branch Repository Path For this article Authentication Type is set to Access Token. This user needs to make sure the Access token is generated and kept handy from the Repository settings page in Bitbucket. Also, the user needs to make sure the Repository permissions are set up properly with Read and Write access on the Repository   BitbucketFlowRegistryClient also needs StandardWebClientServiceProvider in the very first place as a Web Client Service to create an HTTP web client to conne

  In this blog post, you can see multiple Spark3 Integration examples.Spark3 InstallationSpark3 SparkPi Example - Coming soonSpark3 Streaming with Kafka Example - Coming soonSpark3 Structured Streaming with Kafka Example - Coming soonSpark3 HBase Connector ExampleSpark3 Phoenix Connector Example - Coming soonSpark3 Hive Warehouse Connector Example - Coming soonSpark3 Kudu ExampleSpark3 Ozone Example - Coming soonSpark3 Iceberg Example - Coming soonSpark3 Atals Example- Coming soonSpark3 CDE Example - Coming soonSpark3 GPUs Example - Coming soon Spark3 with Custom logging - Coming soonSpark3 Hue IntegrationNote: If you are looking any specific examples other than above you can comment here.

Monitoring Apache NiFiIt's really important to pick some Reporting Tasks to let you know what's happening in Apache NiFi servers. Ambari will send it to your HDF Ambari which will show the results in nice Grafana graphs, charts and tables.You can also monitor disk usage, memory and also send tasks to DataDog, Ganglia and Other Servers. It's also easy to write your own Reporting Task if you need a different one.One of the ways to monitor your Apache NiFi Data Flows is to use the MonitorActivity processor which will create messages that can be sent to your Operations Dashboard, Console or elsewhere.For people doing ChatOps, you can easily push these messages to Slack (there's a processor for that) PutSlack. You could also send a REST call to HipChat or other chat tools. Pretty easy to wrap that up in a custom processor as well.Other Things to MonitorREST END Pointsserver:port/nifi-api/system-diagnosticsSee: https://nifi.apache.org/docs/nifi-docs/rest-api/Logs...nifi/logs/nif

Objective This tutorial is the second article of a three part series. We will walk-through a Apache NiFi CDC flow which uses MySQL bin logs to create a copy of a table and keep the copy in sync with row-level changes to the source. The last article in the series will delve into the finer details of the flow configuration, suggest best practices and highlight potential trouble spots.Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 MySQL 5.7.13 Apache NiFi 1.3.0NiFI CDC Flow Walk ThroughPrerequisites The first article in this tutorial series has been completed and the finalized CDC flow is on your NiFi canvas. Your MySQL instance (with binary logging enabled) is running. A 'users' table exists in your "source" database. Your "copy" database is empty. All controller services are enabled.Flow Overview Here is a quick overview of the CDC flow: 1. CaptureChangeMySQL reads the bin logs to generate FlowFiles (JSON) 2. FlowFiles

One of the most exciting and valuable features (in my opinion) of Cloudera Flow Management 4 (powered by Apache NiFi 2) is the Flow Analysis Rules engine and the various rule implementations you can bring to bear.The Flow Analysis Rules Engine is a built-in capability to Apache NiFi 2 whose mission is to help Flow Administrators enforce Best Practices in flow design. This includes limiting how many resources (threads, memory, disk, etc.) a component can leverage. For example, a Flow Admin may want to restrict all processors to use a maximum of 10 concurrent tasks. One reason for this (especially in large flows) is to prevent a Flow Designer from "hogging" the threads away from other components / teams, or to maximize parallelization across physical CPU cores rather than concurrency (which can be viewed as "apparently parallel" but not truly parallel). The same goes for disk usage that can be enforced by limiting the size of a connection's backpressure settings to prevent too much data

In the final installment of Parameterize All The Things mini-series for the Cloudera Flow Management / Apache NiFi Best Practices cookbook, I will discuss Parameter Providers and how to use them.As we've seen in the previous two posts, we can create Parameter Contexts and manually add key/value pairs to be used as Parameter references in NiFi component properties. But manually maintaining these values can be tedious and this doesn't lend itself to automation. Not only that, but it is possible that the parameter values should not be known by the Flow Designers and instead just used to connect to external systems, such as can be the case for database user passwords, secret keys, etc.To solve this problem, the concept of Parameter Providers was introduced. This capability allows for parameters and their values (both sensitive and plain) to be fetched from an external source. These sources include (but aren't limited to) the following:Environment variablesDatabase tablesPassword vaults suc

In the last article I explained the history of Parameters and Parameter Contexts and how to use them most effectively in your flows. But did you know you can actually swap out Controller Service instances using their UUIDs as parameter values? I will explain why this is valuable as an example:I have a DBConnectionPool (in Apache NiFi) or a PostgreSQLConnectionPool (in Cloudera Flow Management) that points to a PostgreSQL database, and I'm parameterizing all the connection properties using the strategy laid out in part 1 of this series. That's all well and good but what if the folks on the destination side say "we decided to switch to Impala on Cloudera Data Platform because we don't want the cost of managing our PostgreSQL cluster, plus it's integrated with our lakehouse". You can't just change the parameters used in the connection pool because you need an entirely different ControllerService.In Part 1 we were able to swap out configurations for a particular Controller Service, but wha

SummaryIf you are using Java Garbage collection arguments in your settings, when you move to Java 11 or 17 from Java 8 or prior, the settings have changed. Applies toAnyone who has moved to Java 11 or 17 and requires GC logging.InstructionsIn the old GC (Java 8 and prior) logging settings, you would use something similar to:-Xloggc:/var/log/cloudera-scm-server/gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStampsIn Java 11 and 17 this will actually cause an error as these are no longer used.  To get similar output we recommend: -Xlog:gc*,time:file=/var/log/cloudera-scm-server/gc-%t.logwhere gc* will log anything related to GC up to INFO leveltime will give the equivilant data that used to be PrintGCTimeStamps and PrintGCDateStampsfile wil

Minimum User Percentage and User Limit Factor are ways to control how resources get assigned to users within the queues they are utilising. The Min User Percentage is a soft limit on the smallest amount of resources a single user should get access to if they are requesting it. For a specific queue Minimum User Limit Percentage(MULP) is a soft limit on the smallest amount of resources every user will get.This MULP is decided on the basis of how many concurrent users we are expecting to run job on a particular queue. Setting this to 10% is ideal as it will give around 10 users to have minimum of 10% of the queue minimum capacity configured.Setting the config for MULP is also based on the Active and Non Active users. Active users are the users who are requesting for more resources and Non active users are the users who are running their job but not requesting more resources.Generally the Idea is to calculate the MULP for the active users:active-user-limit = max(resource-use

This article is part of the Cloudera Flow Management / Apache NiFi Best Practices Cookbook series and will discuss Best Practices for using Parameters and Parameter Contexts in your flows.HistoryFirst a little background about Parameters: In NiFi 1.x, the original solution for being able to configure various environments (development vs production, e.g.) was Variables. This was a global key/value store of variable names with values associated with them. These variable names could be used in Expression Language expressions as well as certain properties in processors looking specifically for a variable name.This single, global list was not flexible enough to solve many of the configuration challenges of our users. To solve this, Parameters were introduced. They are meant as a replacement for Variables and via Parameter Contexts are much more flexible and powerful due to inheritance (which I will discuss later).ParametersIn general, a parameter is referred to by the following syntax

Welcome! In this series of articles I will address various topics about Cloudera Flow Management (powered by Apache NiFi) in terms of Best Practices about how, when, and why to use the powerful and flexible features of Cloudera Flow Management (CFM) / Apache NiFi. I will use the terms CFM and NiFi interchangeably but will specify when something is specific to Cloudera Flow Management and does not apply to Apache NiFi.These examples and screenshots will be taken both from CFM 2 (powered by Apache NiFi 1.x) and CFM 4 (powered by Apache NiFi 2.x) when prudent to illustrate the features and Best Practices that are common to both CFM 2 and 4, as well as when something is specific to CFM 4 / NiFi 2. Links will be present for articles that have been published.This Best Practices Cookbook will feature the following articles, but please like/subscribe as articles may be added and/or edited at various times. Also if you have any ideas on Best Practices articles that may be helpful to the communi

Objective This tutorial is the final article of a three part series. We will look at the flow components in greater detail to highlight best practices and potential trouble spots when working with CDC in NiFi. Part 1 Part 2Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 MySQL 5.7.13 Apache NiFi 1.3.0CDC Flow Configuration Details, Best Practices & TroubleshootingPrerequisitesIt's helpful to have completed the first two articles in this series(Part 1 ; Part 2) to have a point of reference for the items discussed below. But, this article will hopefully still be useful for those who are interested in some of the nuances of the CaptureChangeMysql, EnforceOrder and PutDatabaseRecord processors.State in CaptureChangeMysqlThe CaptureChangeMySQL processor has a "Distributed Map Cache Client" property which is set to a DistributedMapCacheClientService controller service to store State information and keep it updated. The Distributed