Community Articles

IntroductionApache NiFi 1.0 was recently released and being integrated into Hortonworks Data Flow (HDF) that will be release very soon.In this easy tutorial we will see how we can stream data from CSV format into Hive tables directly and start working on it right away without a single line of coding to set up the streaming. Pre-requisitesIn order to run this tutorial successfully you need to download the Following:NiFi 1.0 or higher, you can download it from hereHDP Sandbox 2.4 or higher, you can download it from hereDownload the Olympics CSV data from the attachment list below. Changing NiFi Port (Optional)Since Ambari and NiFi both uses port 8080, you will have problems starting NiFi if you are running the sandbox and NiFi on the same machine.Once NiFi is downloaded, uncompress it and go to /nifi/conf/nifi.properties and change the port no to 8089 as follows:nifi.web.http.port=8089 Starting NiFi and the SandboxOnce NiFi is downloaded, uncompress and start it using the command:/nifi/

Setting up a new MacBook for Data Engineering is quite a chore. In this guide, we will take you from a fresh macOS installation to running the full suite of Cloudera Streaming Operators (CFM, CSA, CSM) on a local Minikube cluster.   Please note that we are utilizing the Evaluation examples for these services. This is a critical step for any engineer; mastering the foundational configurations is essential before moving into complex, secure, and high-availability production architectures. Let’s get started! My MacBook Setup First, we need the “Swiss Army Knife” of macOS: homebrew. We will use it to install our core CLI tools, container runtimes, and local Kubernetes environment. # Install Homebrew (if not already installed) /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" # Install Core Tools brew install git docker python minikube kubernetes-cli k9s helm # Start Docker Desktop (or Colima) open

Here is the steps to access Ambari UI through Knox. This is tried and tested with Ambari 2.4 and HDP2.51. Make sure Knox is configured properly and it works fine.2. ssh to Knox gateway host and go to /var/lib/knox/data-2.5.3.0-37/services3. download the configurations from https://github.com/apache/knox/tree/v0.11.0/gateway-service-definitions/src/main/resources/services/ambariui/2.2.0/ URL.4. make sure your folder structure should look alike /var/lib/knox/data-2.5.3.0-37/services/ambariui/2.2.0 and should have rewrite.xml and service.xml files5. change the owner/Group permissions to Knox for /var/lib/knox/data-2.5.3.0-37/services/ambariui/ and subdirectory6. Go to Knox configurations Modify "Advanced topology" with below service tag <service> <role>AMBARIUI</role> <url>http://AMBARIHOST:8080</url> </service>7. Restart Knox service.8. You should be able to access Ambari-server UI from the below URLhtt

SummaryIf you have lost your RMON db password and need to retrieve it, in CM GUI it is **** out.  This article will tell you have you can retrieve it in plain text. SymptomsAny customer needing to get into RMON db but not having the password would need to get the password to get in (assuming they do not have DB ROOT)InstructionsTo retrieve the password we can do an API call that gives us the RMON db config info. api/v31/cm/service/roleConfigGroups/mgmt-REPORTSMANAGER-BASE/config an example would be:curl -u admin:admin GET http://ccycloud-1.oteixeira-ubuntu2004.root.comops.site:7180/api/v31/cm/service/roleConfigGroups/mgmt-REPORTSMANAGER-BASE/config​And the results would appear similar to:{ "items" : [ { "name" : "headlamp_database_host", "value" : "ccycloud-1.oteixeira-ubuntu2004.root.comops.site:3306" }, { "name" : "headlamp_database_name", "value" : "reportsmanager9f05b8e6450d2994f37859d1ee3f1967" }, { "name" : "headlamp_database_password", "v

Summary Yarn distributed shell test job fails when running with Ldap group mapping turned on with basic CDP 7.1.9 SP1 cluster using Java 17 JDK.24/12/11 15:21:29 ERROR distributedshell.ApplicationMaster: Error running ApplicationMasterjava.lang.IllegalAccessError: class org.apache.hadoop.security.LdapGroupsMapping (in unnamed module @0x3901d134) cannot access class com.sun.jndi.ldap.LdapCtxFactory (in module java.naming) because module java.naming does not export com.sun.jndi.ldap to unnamed module @0x3901d134at org.apache.hadoop.security.LdapGroupsMapping.<clinit>(LdapGroupsMapping.java:264)Symptoms Applies to  New clusters using CDP 7.1.9 SP1 and Java 17 JDK with Ldap group mapping on.  InstructionsNeed to add In Yarn - Configuration - search for yarn.nodemanager.admin-env and change the NodeManager Group (all of them if you have more then one) to: JDK_JAVA_OPTIONS=--add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.util=ALL-

Apache NiFi 2.X or CFM Version 4.X now provide an ability to use Bitbucket as flow version control in place of NiFi Registry. This new Registry Client is built with the name BitbucketFlowRegistryClient to connect to Bitbucket To set up BitbucketFlowRegistryClient following information is required from the Bitbucket account as a prerequisite; if not present must be created  Workspace Name Repository Name Authentication Type  Default Branch Repository Path For this article Authentication Type is set to Access Token. This user needs to make sure the Access token is generated and kept handy from the Repository settings page in Bitbucket. Also, the user needs to make sure the Repository permissions are set up properly with Read and Write access on the Repository   BitbucketFlowRegistryClient also needs StandardWebClientServiceProvider in the very first place as a Web Client Service to create an HTTP web client to conne

  In this blog post, you can see multiple Spark3 Integration examples.Spark3 InstallationSpark3 SparkPi Example - Coming soonSpark3 Streaming with Kafka Example - Coming soonSpark3 Structured Streaming with Kafka Example - Coming soonSpark3 HBase Connector ExampleSpark3 Phoenix Connector Example - Coming soonSpark3 Hive Warehouse Connector Example - Coming soonSpark3 Kudu ExampleSpark3 Ozone Example - Coming soonSpark3 Iceberg Example - Coming soonSpark3 Atals Example- Coming soonSpark3 CDE Example - Coming soonSpark3 GPUs Example - Coming soon Spark3 with Custom logging - Coming soonSpark3 Hue IntegrationNote: If you are looking any specific examples other than above you can comment here.

Monitoring Apache NiFiIt's really important to pick some Reporting Tasks to let you know what's happening in Apache NiFi servers. Ambari will send it to your HDF Ambari which will show the results in nice Grafana graphs, charts and tables.You can also monitor disk usage, memory and also send tasks to DataDog, Ganglia and Other Servers. It's also easy to write your own Reporting Task if you need a different one.One of the ways to monitor your Apache NiFi Data Flows is to use the MonitorActivity processor which will create messages that can be sent to your Operations Dashboard, Console or elsewhere.For people doing ChatOps, you can easily push these messages to Slack (there's a processor for that) PutSlack. You could also send a REST call to HipChat or other chat tools. Pretty easy to wrap that up in a custom processor as well.Other Things to MonitorREST END Pointsserver:port/nifi-api/system-diagnosticsSee: https://nifi.apache.org/docs/nifi-docs/rest-api/Logs...nifi/logs/nif

Objective This tutorial is the second article of a three part series. We will walk-through a Apache NiFi CDC flow which uses MySQL bin logs to create a copy of a table and keep the copy in sync with row-level changes to the source. The last article in the series will delve into the finer details of the flow configuration, suggest best practices and highlight potential trouble spots.Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 MySQL 5.7.13 Apache NiFi 1.3.0NiFI CDC Flow Walk ThroughPrerequisites The first article in this tutorial series has been completed and the finalized CDC flow is on your NiFi canvas. Your MySQL instance (with binary logging enabled) is running. A 'users' table exists in your "source" database. Your "copy" database is empty. All controller services are enabled.Flow Overview Here is a quick overview of the CDC flow: 1. CaptureChangeMySQL reads the bin logs to generate FlowFiles (JSON) 2. FlowFiles

One of the most exciting and valuable features (in my opinion) of Cloudera Flow Management 4 (powered by Apache NiFi 2) is the Flow Analysis Rules engine and the various rule implementations you can bring to bear.The Flow Analysis Rules Engine is a built-in capability to Apache NiFi 2 whose mission is to help Flow Administrators enforce Best Practices in flow design. This includes limiting how many resources (threads, memory, disk, etc.) a component can leverage. For example, a Flow Admin may want to restrict all processors to use a maximum of 10 concurrent tasks. One reason for this (especially in large flows) is to prevent a Flow Designer from "hogging" the threads away from other components / teams, or to maximize parallelization across physical CPU cores rather than concurrency (which can be viewed as "apparently parallel" but not truly parallel). The same goes for disk usage that can be enforced by limiting the size of a connection's backpressure settings to prevent too much data

In the final installment of Parameterize All The Things mini-series for the Cloudera Flow Management / Apache NiFi Best Practices cookbook, I will discuss Parameter Providers and how to use them.As we've seen in the previous two posts, we can create Parameter Contexts and manually add key/value pairs to be used as Parameter references in NiFi component properties. But manually maintaining these values can be tedious and this doesn't lend itself to automation. Not only that, but it is possible that the parameter values should not be known by the Flow Designers and instead just used to connect to external systems, such as can be the case for database user passwords, secret keys, etc.To solve this problem, the concept of Parameter Providers was introduced. This capability allows for parameters and their values (both sensitive and plain) to be fetched from an external source. These sources include (but aren't limited to) the following:Environment variablesDatabase tablesPassword vaults suc

In the last article I explained the history of Parameters and Parameter Contexts and how to use them most effectively in your flows. But did you know you can actually swap out Controller Service instances using their UUIDs as parameter values? I will explain why this is valuable as an example:I have a DBConnectionPool (in Apache NiFi) or a PostgreSQLConnectionPool (in Cloudera Flow Management) that points to a PostgreSQL database, and I'm parameterizing all the connection properties using the strategy laid out in part 1 of this series. That's all well and good but what if the folks on the destination side say "we decided to switch to Impala on Cloudera Data Platform because we don't want the cost of managing our PostgreSQL cluster, plus it's integrated with our lakehouse". You can't just change the parameters used in the connection pool because you need an entirely different ControllerService.In Part 1 we were able to swap out configurations for a particular Controller Service, but wha

SummaryIf you are using Java Garbage collection arguments in your settings, when you move to Java 11 or 17 from Java 8 or prior, the settings have changed. Applies toAnyone who has moved to Java 11 or 17 and requires GC logging.InstructionsIn the old GC (Java 8 and prior) logging settings, you would use something similar to:-Xloggc:/var/log/cloudera-scm-server/gc.log -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStampsIn Java 11 and 17 this will actually cause an error as these are no longer used.  To get similar output we recommend: -Xlog:gc*,time:file=/var/log/cloudera-scm-server/gc-%t.logwhere gc* will log anything related to GC up to INFO leveltime will give the equivilant data that used to be PrintGCTimeStamps and PrintGCDateStampsfile wil

Minimum User Percentage and User Limit Factor are ways to control how resources get assigned to users within the queues they are utilising. The Min User Percentage is a soft limit on the smallest amount of resources a single user should get access to if they are requesting it. For a specific queue Minimum User Limit Percentage(MULP) is a soft limit on the smallest amount of resources every user will get.This MULP is decided on the basis of how many concurrent users we are expecting to run job on a particular queue. Setting this to 10% is ideal as it will give around 10 users to have minimum of 10% of the queue minimum capacity configured.Setting the config for MULP is also based on the Active and Non Active users. Active users are the users who are requesting for more resources and Non active users are the users who are running their job but not requesting more resources.Generally the Idea is to calculate the MULP for the active users:active-user-limit = max(resource-use

This article is part of the Cloudera Flow Management / Apache NiFi Best Practices Cookbook series and will discuss Best Practices for using Parameters and Parameter Contexts in your flows.HistoryFirst a little background about Parameters: In NiFi 1.x, the original solution for being able to configure various environments (development vs production, e.g.) was Variables. This was a global key/value store of variable names with values associated with them. These variable names could be used in Expression Language expressions as well as certain properties in processors looking specifically for a variable name.This single, global list was not flexible enough to solve many of the configuration challenges of our users. To solve this, Parameters were introduced. They are meant as a replacement for Variables and via Parameter Contexts are much more flexible and powerful due to inheritance (which I will discuss later).ParametersIn general, a parameter is referred to by the following syntax

Welcome! In this series of articles I will address various topics about Cloudera Flow Management (powered by Apache NiFi) in terms of Best Practices about how, when, and why to use the powerful and flexible features of Cloudera Flow Management (CFM) / Apache NiFi. I will use the terms CFM and NiFi interchangeably but will specify when something is specific to Cloudera Flow Management and does not apply to Apache NiFi.These examples and screenshots will be taken both from CFM 2 (powered by Apache NiFi 1.x) and CFM 4 (powered by Apache NiFi 2.x) when prudent to illustrate the features and Best Practices that are common to both CFM 2 and 4, as well as when something is specific to CFM 4 / NiFi 2. Links will be present for articles that have been published.This Best Practices Cookbook will feature the following articles, but please like/subscribe as articles may be added and/or edited at various times. Also if you have any ideas on Best Practices articles that may be helpful to the communi

Objective This tutorial is the final article of a three part series. We will look at the flow components in greater detail to highlight best practices and potential trouble spots when working with CDC in NiFi. Part 1 Part 2Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 MySQL 5.7.13 Apache NiFi 1.3.0CDC Flow Configuration Details, Best Practices & TroubleshootingPrerequisitesIt's helpful to have completed the first two articles in this series(Part 1 ; Part 2) to have a point of reference for the items discussed below. But, this article will hopefully still be useful for those who are interested in some of the nuances of the CaptureChangeMysql, EnforceOrder and PutDatabaseRecord processors.State in CaptureChangeMysqlThe CaptureChangeMySQL processor has a "Distributed Map Cache Client" property which is set to a DistributedMapCacheClientService controller service to store State information and keep it updated. The Distributed

Objective This tutorial walks you through the process of using the CaptureChangeMySQL, EnforceOrder and PutDatabaseRecord processors in Apache NiFi to replicate a database using MySQL event logs. This article is the first of a three part series. We will setup the demo environment and components, create a multi-row MySQL table (which generates binary log data) and import a NiFi CDC flow template. The second part of the series will walk-through the CDC flow showing how the MySQL bin logs can be used to create a copy of the table and how the flow keeps the copy in sync with any row-level changes to the source table. The last article in the series will delve into the finer details of the flow configuration, suggest best practices and highlight potential trouble spots.Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 MySQL 5.7.13 Apache NiFi 1.3.0 Note: CaptureChangeMySQL, EnforceOrder and PutDatabaseRecord processors were introduc

Requirement introduction Azure PostgresDB is a PaaS hosted on Azure. Azure hosts PaaS services in an Azure backend subscription. The resources created in the customer’s subscription are actually descriptions and metadata of the actual compute resources in the Azure backend subscription.  When the Azure backend subscription reaches its capacity limit, Azure will create new resources in a new Azure backend subscription. However, existing PaaS resources will remain in the old Azure backend subscription. When a CDP  user stops a CDP environment, the compute resources for the PostgresDB will be deallocated from the backend Azure subscription, and have to compete for compute resources when being restarted. Suppose it cannot get resources from the backend Azure subscription, as it is already out of capacity. In that case, it cannot be started, and consequently blocks the environment from being started.  The resolution provided by Microsoft is to back up and restore the database. The

Pulling Change Data Capture (aka CDC) logs off a database can be challenging, but is made easier with tools like Debezium and Kafka Connect.  It can still be challenging, but Cloudera's Streams Messaging Manager has an interface to help simplify the setup process.   At its core, Kafka Connect simply takes data sources (think:  relational database, message queue, syslogs, etc.) and publishes that data to a kafka topic.  It can also read data from a kafka topic and sink that data to another location (think:  S3, HTTP endpoint, database, etc).   It connects sources to sinks using kafka as the intermediate layer.   For our purposes, we will set up a connection to read CDC logs off a MySQL instance and publish that data to kafka. Pre-requisites:CDP Public Cloud environmentA Streams Messaging data hub (light duty is fine)A MySQL instance that you can administer & restart Helpful documents:Installing MySQL on Linux Configuri

This article provides detailed steps about Encrypting and Decrypting files using the Pretty Good Privacy (PGP)  algorithm in NiFi Steps involve : Setting up GPG private key and Public key Creating NiFi flow  Setting up GPG private key and Public key Set up the GPG home directory export GNUPGHOME=/tmp/gnupg​ Generate a private and public key  with a user IDgpg --full-generate-keygpg (GnuPG) 2.2.20; Copyright (C) 2020 Free Software Foundation, Inc.This is free software: you are free to change and redistribute it.There is NO WARRANTY, to the extent permitted by law.Please select what kind of key you want:(1) RSA and RSA (default)(2) DSA and Elgamal(3) DSA (sign only)(4) RSA (sign only)(14) Existing key from cardYour selection? 1RSA keys may be between 1024 and 4096 bits long.What keysize do you want? (2048) 2048Requested keysize is 2048 bitsPlease specify how long the key should be valid.0 = key does not expire<n> = key expires in n days<n>w =

Cloudera Data Engineering offers the integration with Hive Warehouse Connector (HWC) to access and manage Hive-managed tables from Cloudera Data Engineering Spark. The HWC dependencies are built into the Cloudera Data Engineering product.HWC can be used to "Read Data from Hive" and "Write Data to Hive" through Cloudera Data Engineering Spark.The data can be read and written in the following ways using HWC. Each of these read and write modes has its own advantages and limitations. Hence, this needs to be chosen carefully based on the use case.Read modes: DIRECT_READER_V2JDBC_CLUSTERSECURE_ACCESSWrite modes: HIVE_WAREHOUSE_CONNECTORDATAFRAME_TO_STREAMSTREAM_TO_STREAMNote: Cloudera recommends using the "SECURE_ACCESS" read mode to enforce Fine-Grained Access Control (FGAC) through Ranger authorization in the production environment with large workloads.HWC provides a wide range of functionalities to Cloudera Data Engineering, which includes: CREATE/DROP databasesCREATE Exter

This guide is Part 2 of our series on GRPO-based fine-tuning. It assumes you are already familiar with the core concepts of reasoning models and GRPO (see Part 1 Understanding Reasoning Models with GRPO: A Conceptual Introduction for Building your own Medical Reasoning Model ), and you are ready to apply them in practice using Python. All code is provided in a reproducible notebook, with detailed explanations and references to help you get started.Abstract: In this guide, we’ll walk step by step through fine tuning a large language model on a medical reasoning dataset from Hugging Face, using Group Relative Policy Optimization (GRPO). In the following sections, you will see exactly how the technical setup of GRPO works, how the data is structured, how to create custom reward functions, and how to test, evaluate, and save the medical reasoning model. By the end, you will have a working GRPO pipeline for building your own medical reasoning model that produces <reasoning>..</reas

1. IntroductionProblem:Most readily-available Medical AI models lack structured reasoning, which significantly limits their reliability and trustworthiness in critical clinical decision-making. GRPO offers a robust solution for building reasoning Medical AI models by reinforcing logical step-by-step explanations.Takeaway:This post explains how enterprises can fine-tune Large Language Models (LLMs)  with reinforcement learning (RL) using GRPO for more structured, explainable AI, demonstrating its application with a medical reasoning use case.2. The Business Case Why should enterprises care?Reasoning is critical in high-stakes industries like healthcare, finance, and legal sectors.  It provides better transparency, explainability, and tracing on how models generate their outputs, building crucial trust and facilitating accountability. Why GRPO?GRPO efficiently delivers advanced reasoning capabilities. It significantly reduces  compute requirements, cutting them b

Abstract: CDP is using many managed cloud services from different CSPs as underlying infrastructure. The DNS configuration in a typical CDP onboarding is critical to a successful deployment. This article introduces some common DNS setup issues with AWS and corresponding solutions.AWS services with private endpoints used by CDP Most enterprise AWS consumers are using HUB-SPOKE network architecture with custom private DNS. The Custom private DNS usually resides on HUB VPC. Correct DNS configuration needs to be created for a successful deployment.AWS managed services CDP is usingRDSElastic Kubernetes ServiceElastic File ServiceIn a typical HUB-Spoke network, the custom private DNS server is usually deployed in the HUB network. There are several other DNS servers that are being used. AWS Public DNS: Hosted by AWS on internetAWS Default Private DNS: One per VPC. At 169.254.169.253 Different AWS services use different DNS for its domain registration.  AWS RDS create an en