Resources

Objective This tutorial walks you through the process of using the CaptureChangeMySQL, EnforceOrder and PutDatabaseRecord processors in Apache NiFi to replicate a database using MySQL event logs. This article is the first of a three part series. We will setup the demo environment and components, create a multi-row MySQL table (which generates binary log data) and import a NiFi CDC flow template. The second part of the series will walk-through the CDC flow showing how the MySQL bin logs can be used to create a copy of the table and how the flow keeps the copy in sync with any row-level changes to the source table. The last article in the series will delve into the finer details of the flow configuration, suggest best practices and highlight potential trouble spots.Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 MySQL 5.7.13 Apache NiFi 1.3.0 Note: CaptureChangeMySQL, EnforceOrder and PutDatabaseRecord processors were introduc

Requirement introduction Azure PostgresDB is a PaaS hosted on Azure. Azure hosts PaaS services in an Azure backend subscription. The resources created in the customer’s subscription are actually descriptions and metadata of the actual compute resources in the Azure backend subscription.  When the Azure backend subscription reaches its capacity limit, Azure will create new resources in a new Azure backend subscription. However, existing PaaS resources will remain in the old Azure backend subscription. When a CDP  user stops a CDP environment, the compute resources for the PostgresDB will be deallocated from the backend Azure subscription, and have to compete for compute resources when being restarted. Suppose it cannot get resources from the backend Azure subscription, as it is already out of capacity. In that case, it cannot be started, and consequently blocks the environment from being started.  The resolution provided by Microsoft is to back up and restore the database. The

Hi, I am changing my organization. Currently i have my organization email id in my account which is also my user name. I have to change it to a new email address. How can i change it? Can you please help me change it?

Pulling Change Data Capture (aka CDC) logs off a database can be challenging, but is made easier with tools like Debezium and Kafka Connect.  It can still be challenging, but Cloudera's Streams Messaging Manager has an interface to help simplify the setup process.   At its core, Kafka Connect simply takes data sources (think:  relational database, message queue, syslogs, etc.) and publishes that data to a kafka topic.  It can also read data from a kafka topic and sink that data to another location (think:  S3, HTTP endpoint, database, etc).   It connects sources to sinks using kafka as the intermediate layer.   For our purposes, we will set up a connection to read CDC logs off a MySQL instance and publish that data to kafka. Pre-requisites:CDP Public Cloud environmentA Streams Messaging data hub (light duty is fine)A MySQL instance that you can administer & restart Helpful documents:Installing MySQL on Linux Configuri

This article provides detailed steps about Encrypting and Decrypting files using the Pretty Good Privacy (PGP)  algorithm in NiFi Steps involve : Setting up GPG private key and Public key Creating NiFi flow  Setting up GPG private key and Public key Set up the GPG home directory export GNUPGHOME=/tmp/gnupg​ Generate a private and public key  with a user IDgpg --full-generate-keygpg (GnuPG) 2.2.20; Copyright (C) 2020 Free Software Foundation, Inc.This is free software: you are free to change and redistribute it.There is NO WARRANTY, to the extent permitted by law.Please select what kind of key you want:(1) RSA and RSA (default)(2) DSA and Elgamal(3) DSA (sign only)(4) RSA (sign only)(14) Existing key from cardYour selection? 1RSA keys may be between 1024 and 4096 bits long.What keysize do you want? (2048) 2048Requested keysize is 2048 bitsPlease specify how long the key should be valid.0 = key does not expire<n> = key expires in n days<n>w =

Cloudera Data Engineering offers the integration with Hive Warehouse Connector (HWC) to access and manage Hive-managed tables from Cloudera Data Engineering Spark. The HWC dependencies are built into the Cloudera Data Engineering product.HWC can be used to "Read Data from Hive" and "Write Data to Hive" through Cloudera Data Engineering Spark.The data can be read and written in the following ways using HWC. Each of these read and write modes has its own advantages and limitations. Hence, this needs to be chosen carefully based on the use case.Read modes: DIRECT_READER_V2JDBC_CLUSTERSECURE_ACCESSWrite modes: HIVE_WAREHOUSE_CONNECTORDATAFRAME_TO_STREAMSTREAM_TO_STREAMNote: Cloudera recommends using the "SECURE_ACCESS" read mode to enforce Fine-Grained Access Control (FGAC) through Ranger authorization in the production environment with large workloads.HWC provides a wide range of functionalities to Cloudera Data Engineering, which includes: CREATE/DROP databasesCREATE Exter

This guide is Part 2 of our series on GRPO-based fine-tuning. It assumes you are already familiar with the core concepts of reasoning models and GRPO (see Part 1 Understanding Reasoning Models with GRPO: A Conceptual Introduction for Building your own Medical Reasoning Model ), and you are ready to apply them in practice using Python. All code is provided in a reproducible notebook, with detailed explanations and references to help you get started.Abstract: In this guide, we’ll walk step by step through fine tuning a large language model on a medical reasoning dataset from Hugging Face, using Group Relative Policy Optimization (GRPO). In the following sections, you will see exactly how the technical setup of GRPO works, how the data is structured, how to create custom reward functions, and how to test, evaluate, and save the medical reasoning model. By the end, you will have a working GRPO pipeline for building your own medical reasoning model that produces <reasoning>..</reas

1. IntroductionProblem:Most readily-available Medical AI models lack structured reasoning, which significantly limits their reliability and trustworthiness in critical clinical decision-making. GRPO offers a robust solution for building reasoning Medical AI models by reinforcing logical step-by-step explanations.Takeaway:This post explains how enterprises can fine-tune Large Language Models (LLMs)  with reinforcement learning (RL) using GRPO for more structured, explainable AI, demonstrating its application with a medical reasoning use case.2. The Business Case Why should enterprises care?Reasoning is critical in high-stakes industries like healthcare, finance, and legal sectors.  It provides better transparency, explainability, and tracing on how models generate their outputs, building crucial trust and facilitating accountability. Why GRPO?GRPO efficiently delivers advanced reasoning capabilities. It significantly reduces  compute requirements, cutting them b

Abstract: CDP is using many managed cloud services from different CSPs as underlying infrastructure. The DNS configuration in a typical CDP onboarding is critical to a successful deployment. This article introduces some common DNS setup issues with AWS and corresponding solutions.AWS services with private endpoints used by CDP Most enterprise AWS consumers are using HUB-SPOKE network architecture with custom private DNS. The Custom private DNS usually resides on HUB VPC. Correct DNS configuration needs to be created for a successful deployment.AWS managed services CDP is usingRDSElastic Kubernetes ServiceElastic File ServiceIn a typical HUB-Spoke network, the custom private DNS server is usually deployed in the HUB network. There are several other DNS servers that are being used. AWS Public DNS: Hosted by AWS on internetAWS Default Private DNS: One per VPC. At 169.254.169.253 Different AWS services use different DNS for its domain registration.  AWS RDS create an en

The NiFi community maintains a NiFi Docker image that comes very handy when you want to play with NiFi on your laptop. When you develop flows to run in a NiFi cluster with 2 or more nodes, though, there are things that you need to check and test differently from a single-node NiFi cluster. Recently, @Arqui shared a docker-compose configuration in this forum question that creates a 2-node NiFi cluster on Docker. The compose configuration still had a few issues to be ironed, though, and there were some manual steps you still had to perform before launching the cluster, like creating the TLS certificates for it. I found the idea interesting and spent some time working out the missing details. I also automated the TLS certificate creation and added a load-balancer to the compose configuration. Now, to use your own NiFi sanbox cluster, all you need to do is this: Download the docker-compose.yml file (see repo link below)Run "docker compose up -d"Connect to your

This guide provides a step-by-step approach to extracting data from SAP S/4HANA via OData APIs, processing it using Apache NiFi in Cloudera Data Platform (CDP), and storing it in an Iceberg-based Lakehouse for analytics and AI workloads. 1. Introduction1.1 Why Move SAP S/4HANA Data to a Lakehouse?SAP S/4HANA is a powerful ERP system designed for transactional processing, but it faces limitations when used for analytics, AI, and large-scale reporting:Performance Impact:Running complex analytical queries directly on SAP can degrade system performance.Limited Scalability:SAP systems are not optimized for big data workloads (e.g., petabyte-scale analytics).High Licensing Costs:Extracting and replicating SAP data for analytics can be expensive if done inefficiently.Lack of Flexibility:SAP’s data model is rigid, making it difficult to integrate with modern AI/ML tools.A Lakehouse architecture (built on Apache Ice

A resource in Cloudera Data Engineering is a named collection of files used by a job or a session. Resources can include application code, configuration files, custom Docker images, and Python virtual environment specifications (requirements.txt).CDE 1.24 introduces granular ACL's to prevent developers from utilizing and modifying resources in use by other users' Spark and Airflow jobs. In this article you will learn how to use the aforementioned ACL's with Jobs and Resources using the CDE CLI.Background Info on CDE and the CLICDE is the Cloudera Data Engineering Service, a containerized managed service for Cloudera Data Platform designed for Large Scale Batch and Streaming Pipelines with Spark, Airflow and Iceberg. It allows you to submit batch jobs to auto-scaling virtual clusters. As a Cloud-Native service, CDE enables you to spend more time on your applications, and less time on infrastructure.CDE allows you to create, manage, and schedule Apache Spark jobs without the overhead of

Cruise Control is an open-source solution to automate the dynamic workload rebalance and self-healing of large-scale Apache Kafka clusters. With the widespread adoption of Apache Kafka, many companies now maintain Kafka clusters with hundreds or even thousands of brokers. Operating large-scale clusters may lead to frequent broker failures and a substantial burden in redistributing workload. Cruise Control was designed to solve these operational scalability challenges.What is Cloudera Streams Messaging - Kubernetes Operator? Cloudera Streams Messaging - Kubernetes Operator brings enterprise-grade Kafka deployments to existing Kubernetes infrastructures. Cloudera Streams Messaging - Kubernetes Operator allows flexible, agile, and rapid deployment as well as scaling for variable workloads. This solution enables the deployment of Kafka and related components on existing, shared Kubernetes infrastructure, eliminating the need for dedicated infrastructureCruise Control is also part of t

What is Cloudera Global Support?Cloudera Global Support provides 24x7 customer assistance through a globally distributed team, combining deep technical expertise with proactive and predictive support capabilities to help customers successfully run Cloudera products.What issues are covered under Cloudera Support?Cloudera Support focuses on break/fix issues, including:Product crashes or failuresConfiguration issuesUnexpected product behaviorErrors during installation, upgrade, or deploymentBug fixes, patches, and workaroundsFor customer onboarding meant for new customers and/or new contacts within existing accounts to understand how Cloudera Support works, please click here.What is not covered under Cloudera Support ?Support does not include architecture design, performance tuning, training, custom solution development, or third-party system debugging. These requests are handled through Account Teams or Professional Services.How to access the support portal ?To access the support portal

With the addition of Python to Apache NiFi 2.0 and Cloudera Flow Management 4, developers now have a new way to rapidly develop processors and can leverage Python libraries for FlowFile processing, which is an exciting new feature. But what about NiFi’s scripting capabilities using Jython? In this post I will explore the history of Jython and Python in the Apache NiFi project, their capabilities, and the current state of Python and Jython in both Apache NiFi 2 and Cloudera Flow Management 4. NiFi’s Scripting Capabilities: Apache NiFi 0.5.0 introduced the ExecuteScript processor, allowing developers to interact with the NiFi SDK using several JVM-based scripting languages such as Groovy, Javascript, Lua, Clojure, and Jython, a JVM-based version of Python. The Jython language is compatible with Python,  so any  script you write in pure Python should work in ExecuteScript using Jython. “Pure” Python means the entire code is written in the Python language rather than ot

Spark Streaming Graceful shutdown 1. Introduction  In the first part of the blog post, we discussed the importance of graceful shutdown in Spark Streaming applications. We also saw different ways to shutdown a Spark Streaming application gracefully, including: Explicitly calling the JVM Shutdown Hook in the driver program. Using spark.streaming.stopGracefullyOnShutdown = true setting. In the second part, we will discuss a better way to gracefully shutdown the streaming application using other approaches.   2. Use an external approach to control internal program shutdown This approach allows an external process to control the shutdown of the internal program by sending a marker record, which is detected by the internal program. The internal program can then initiate a graceful shutdown procedure, ensuring that any unsaved data is saved and any resources are properly released before exiting. Benefits: It is more reliable than relying on the internal application to shut down

Introduction:In today's digital age, data security and compliance are paramount. Organizations handling sensitive information, especially in cloud environments, need robust tools to monitor and audit user activities. Cloudera offers a comprehensive auditing system to keep track of who's doing what in your environment. In this article, we'll delve into the world of audits in CDP Cloudera, explaining why they are essential and how to use them effectively.Why Audits Matter:Enhancing Security: Audits play a pivotal role in maintaining the security of your CDP Cloudera environment. By keeping a detailed log of user activities, audits help you identify suspicious or unauthorized actions promptly.Compliance Requirements: Many industries and organizations have stringent regulatory requirements. Audits help organizations adhere to these regulations by providing a clear record of data access and modifications.Investigation and Troubleshooting: When issues arise, audits serve as valuable tools fo

Objective In this article, you will create a CDE Spark Job with dependencies in multiple CDE Repositories. On CDE and the CLI CDE is the Cloudera Data Engineering Service, a containerized managed service for Cloudera Data Platform designed for large-scale batch and Streaming Pipelines with Spark, Airflow, and Iceberg. It allows you to submit batch jobs to auto-scaling virtual clusters. As a Cloud-Native service, CDE enables you to spend more time on your applications and less on infrastructure. CDE allows you to create, manage, and schedule Apache Spark jobs without the overhead of creating and maintaining Spark clusters. With CDE, you define virtual clusters with a range of CPU and memory resources, and the cluster scales up and down as needed to run your Spark workloads, helping to control your cloud costs. CDE provides a command-line interface (CLI) client. You can use the CLI to create and update jobs, view job details, manage job resources, run jobs, etc. Requirements The followin

Introduction Data Scientists are consistently looking to augment their data sources with additional datasets either to incorporate new features, refresh the latest data, or validate existing features. In Part 1 of this two-part article, we have covered how we can ingest data using Rest API endpoints.  In Part 2, we will explore how you can add data stored in an external file system into Cloudera Data Lake.  Specifically, we will ingest data from external S3 buckets of a publicly available dataset into a Cloudera Data Lake. Data Source  We will use the the publicly available Air Quality data made available by OpenAQ for this article. To know more about OpenAQ data available for developers , you can visit their website.  Pre-requisite  Access to the Cloudera AI platform  OpenAQ API Key ( available for free, on registration at their website www.openaq.org)  We plan to use a dataset from OpenAQ data source for getting Air Quality data

Introduction Data Scientists are consistently looking to augment their data sources with additional datasets either to incorporate new features, refresh the latest data, or validate existing features. In this 2 part article, we will explore two approaches to augment external data in the Cloudera Data Lake. We will explore how REST Catalog and external S3 buckets of a publicly available dataset can be accessed within ClouderaAI to ingest data.  This is  Part 1 of a two-part article. The Part 2 covers data ingestion from an external file system (s3). Data Source  In this article, we will use the publicly available Air Quality data made available by OpenAQ. Visit their website to learn more about OpenAQ data available for developers.  Pre-requisite  Access to the Cloudera AI platform  OpenAQ API Key ( available for free, on registration at their website www.openaq.org)  We plan to use a dataset from OpenAQ data source for getting Air Quality data for ci

Objective Deploying Dask Cuda Clusters on Kubernetes for distributed GPU workloads can require time, effort, and money, especially at Enterprise scale. You'd have to set up a Kubernetes cluster with GPU support, manage docker images, and dask cuda workers via complex deployment files. Cloudera AI simplifies distributed compute use cases in the context of Machine Learning and AI. In this article, you will learn how to quickly deploy a Dask Cuda Cluster in Cloudera AI using the cmlextensions library. Background Information Dask CUDA clusters enable scalable parallel computing on NVIDIA GPUs by leveraging Dask, a flexible parallel computing framework, in conjunction with the power of CUDA for GPU acceleration. These clusters allow users to distribute computation across multiple GPUs, significantly speeding up data processing and machine learning workflows, particularly for tasks involving large datasets or complex algorithms. Cloudera AI is a suite of artificial intelligence and machine l

Hive-Kafka Integration stepsThis procedure is well suited for a Kerberos and auto TLS-enabled cluster.To begin with please create client.properties files with entries as shown below in the bootstrap server(Active Controller Node check this from cm-->kafka--->instances-->xxxx.net.com(activecontroller):security.protocol=SASL_SSLsasl.mechanism=GSSAPIsasl.kerberos.service.name=kafkassl.truststore.location=/var/run/cloudera-scm-agent/process/71-kafka-KAFKA_CONNECT/cm-auto-global_truststore.jksssl.truststore.password=xxxxxxxssl.keystore.location=/var/run/cloudera-scm-agent/process/71-kafka-KAFKA_CONNECT/cm-auto-host_keystore.jksssl.keystore.password=xxxxxxxsasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required useTicketCache=true;2. Also it is important to export the Jaas.conf file from the respective path as given below.export KAFKA_OPTS="-Djava.security.auth.login.config=/tmp/jaas.conf"Topic creation in Kafka (please get a valid tkt if it is a Kerberos-enabled clus

In YARN, resource allocation discrepancies can occur due to the way resource calculation is handled. By default, resource availability is determined based on available memory.However, when CPU scheduling is enabled, resource calculation considers both available memory and vCores. As a result, in some scenarios, nodes may appear to allocate more vCores than the configured limit while simultaneously displaying lower available resources.This happens due to the way YARN dynamically assigns vCores based on workload demands rather than strictly adhering to preconfigured limits. Additionally, in cases where CPU scheduling is disabled, YARN relies solely on memory-based resource calculation. This may lead to negative values appearing in the YARN UI, which can be safely ignored, as they do not impact actual resource utilization.

Introduction This post explains the process of submitting Spark jobs on CDP Public Cloud Data Hub to the Livy server and provides a sample wrapper script for job submission. To understand the Livy Spark submit process, please refer to this post and understand how to arrive at the job configuration. The following steps submit the Spark Pi job from the spark_examples*.jar on S3. Steps Get the Livy endpoint for Data hub from the CDP control plane Ensure that you have the workload password set for the environment Download the Python wrapper script using the following command: wget https://raw.githubusercontent.com/karthikeyanvijay/cdp-publiccloud/master/datahub-scripts/livy-cdp-spark-submit/cdp_spark_submit.py Edit the script to modify the Livy endpoint, workload username, and password Copy the spark sample jar to the S3 hdfs dfs -cp /opt/cloudera/parcels/CDH-7.2.7-1.cdh7.2.7.p6.11615609/lib/spark/examples/jars/spark-examples_2.11-2.4.5.7.2.7.6-2.jar \ s3a://vkarthikeyan/ &

Cloudera on cloud offers a scalable, flexible, and cost-efficient solution for data management and analytics in the public cloud. It integrates data applications into a unified platform, enabling seamless data consolidation, advanced analytics, and machine learning. With its pay-as-you-go pricing model and managed services, Cloudera on cloud optimizes costs and simplifies operations. It enhances security and compliance, supports hybrid and multi-cloud environments, and provides tools for real-time data processing and self-service analytics. Cloudera on cloud empowers organizations to manage data efficiently while leveraging the latest cloud technologies.In comparison to Cloudera on premises, costs in public cloud can escalate quickly if not carefully monitored, a common challenge with all cloud workloads. However, with effective tracking and cost optimization strategies, public cloud adoptions can offer significant benefits in performance, scalability and cost savings over traditional