Resources

LIMITATIONS OF TEMPLATES If a user wants to change the flows using templates, i.e., downloading templates and uploading them back, then the following issues will occur:  The template will not preserve the state. It is not supposed to. The template is just a blueprint of the components, not their data. New controller services will be created instead of using already existing ones, which is expected behavior in templates. Templates will be unsupported in CFM 4.0/ NiFi 2.0 and there are not active improvements being done to them. FLOW DEFINITION The better solution in this case will be to use flow definitions instead of templates. The steps to use flow definitions are:  Right-click on Process Group > Click on Download Flow Definition > select the option to download flow JSON with or without external controller services. “External” here means services referenced inside the PG, but existing at a higher level. Then, if you import that flow definition (JSON) that has refer

NiFi JSON to DDL Custom ProcessorJava ClassJUnitThis is further enhanced version of the idea started here: https://community.hortonworks.com/articles/154957/converting-json-to-sql-ddl.html?es_p=6294995There was some discussion on linkedin about the previous article being a good processor, so I decided to do that. This is pretty basic, but it handles most types okay. Date and number processing is a bit hacky, but guesses some types.To install, copy the NAR file that you build or download from Github to your NiFi/lib directories and restart those servers.Add the New Processor to Your FlowConfigure the Processor with a table type (that is ignored in this version)Configure the Processor with a table name (this is important)JsonToDDLProcessor Generated DocsI configured my table name to be the filename without an extension for JSONOutput in NiFiExample FlowEnhancements In Consideration: Apache OpenNLPApache TikaAttribute Cleaner EnhancementDeep Learning for Determining TypesMachine Lea

Upgrade your COD cluster without downtime Released on August 10, 2023 You can now upgrade the Cloudera Runtime version and the operating system version of the database that uses HDFS and Object Store without Ephemeral Storage, in a rolling restart mode. Previously, while performing a Cloudera Runtime upgrade, users experienced HBase or Phoenix service downtime provided by COD. You can now avoid a service outage of a COD cluster while performing a Cloudera Runtime upgrade. This feature ensures continuous service availability. To upgrade the Cloudera Runtime version, use the --rolling-upgrade | --no-rolling-upgrade option on CDP CLI with the upgrade-database command. cdp opdb upgrade-database --environment <environment-name> --database <database-name> --runtime <runtime-version> [--rolling-upgrade | --no-rolling-upgrade] To upgrade the operating system version, use the --os-upgrade-only option along with the --rolling-upgrade option in the upgrade-database command. cdp

Short Description:This Tutorial describes how to add fields,removing not required fields and change values of fields in flowfile.ArticleIntroductionUsing UpdateRecord processor we are can update the contents of flowfile.Overview of article:Below sections describes the changes that are going to happen to the input flowfile content vs output flowfile contents.Input:input Json record as follows and having 6 fields/elements in it.[ { "id" : 1, "name" : "foo", "age" : 20, "state" : "FLORIDA", "ts_milli" : 1525526792098, "ts" : "2018-05-03 10:10:10.123"}]Expected Output:I'm writing output in Avro Format but for viewing purpose i have converted the output flowfile to json.[ { "id" : "1", "rename_id" : "1", //newly added field based on id "state" : "florida", //changed the value of field "rename_state" : "FLORIDA", //newly added field based on state "unique_id" : "1-416425265990923-FLORIDA-aab46988-6a27-4008-9a4d-65655abe9c3c",//id-filename-state-UUID "ts_milli" : 1525526792098,

Hive 3.0 introduced an option to re-attempt a failed Hive query, in case the first run fails. It would only make sense if we fixed whatever was the issue in the previous run. We'll discuss the ways to configure this once without having to intervene after each failure event.The following Hive property enables query re-execution. This should be enabled out of the box.hive.query.reexecution.enabled=true;hive.query.reexecution.strategies=overlay,reoptimize,recompile_without_cbo,reexecute_lost_am;Re-execution strategies:OverlayUsing this method, we can set a Hive property that should be applied on the re-execution. It works by adding a configuration subtree as an overlay to the actual hive settings(reexec.overlay.*).set reexec.overlay.{hive_property}=new_valueEvery hive setting which has a prefix of "reexec.overlay" will be set for all re-executions.e.g.In case our Hive queries fail with OOM while performing Map Joins, which could occur when we do not have correct stats for the tables, we c

Objective: Describe a process to perform DI check using spark JDBC and also oozie spark action.Most of the ETL pipelines involve DI checks between source system and ETL environment to ensure that the record counts match, and also sum/average of certain significant columns is compared on both source & ETL environments. We will see how DI check can be performed and results are stored for audit purpose using spark JDBC. We will also talk about how to prevent storing clear text password in order to make JDBC call using spark.The following components need to be written in order to accomplish this. PySpark code that invokes the Java class and retrieves clear text password at runtime & does DI check.Java class to decrypt the password using hadoop credentials java APIOozie spark action to trigger the spark code that performs DI check (counts)PySpark Code for DI Check: This python module calls a java method to decrypt the password and uses it in the spark jdbc call to get the row count

CDE is the Cloudera Data Engineering Service, a containerized managed service for the Cloudera Data Platform designed for Large Scale Batch Pipelines with Spark, Airflow, and Iceberg. It allows you to submit batch jobs to auto-scaling virtual clusters. As a Cloud-Native service, CDE enables you to spend more time on your applications and less time on infrastructure. CDE allows you to create, manage, and schedule Apache Spark jobs without the overhead of creating and maintaining Spark clusters. With CDE, you define virtual clusters with a range of CPU and memory resources, and the cluster scales up and down as needed to run your Spark workloads, helping to control your cloud costs. Cloudera Data Engineering provides a command-line interface (CLI) client. The CDE CLI doesn't just allow you to create CDE Spark Jobs with syntax that is nearly identical to that of a Spark Submit. The CDE CLI also offers you invaluable advantages as part of your daily Spark Pipeline development and managemen

High level Problem ExplanationThe main issue impacting Namenode performance is the global FSNamesystem lock during read and write operations. Handling tasks in larger directories, especially those involving millions of files, can lead to high lock times and overall slowness in operations like extensive file listings, retrieving block locations, handling large deletions, performing DU (Usage) and BDR snapshot activities. ------------ (Global lock - NameSystem) -------- | | | Directory set (InodeMap) Block set(BlockMap) Cluster information(DatanodeManager)Detailed Technical ExplanationRPC request #getListing:DFSClient initiates getList for the given path, invoking DistributedFileSystem#listStatus. It is important to note that this process isn't atomic, and the directory entries may be retrieved from the NameNode multiple times.DFSClient retrieves the initial batch of entr

NiFi User Authentication with LDAP in HW SandboxPrerequisite 1) Assuming you already have NiFi-0.4.0/HDF-1.1 downloaded on your HW Sandbox, else execute below after ssh connectivity to sandbox is established.cd /opt/wget http://public-repo-1.hortonworks.com/HDF/1.1.0.0/nifi-1.1.0.0-10-bin.zipunzip nifi-1.1.0.0-10-bin.zip2) Make sure Knox is installed on your sandbox and demo LDAP is started via AmbariSteps:1. I already created Certification Authorities and client certificates at www.tinycert.orgHow to create certificates as well configurations can be found in below article:https://community.hortonworks.com/content/kbentry/886/securing-nifi-step-by-step.htmlIf you are too lazy to create them, try with mine 🙂 [Attached as certificates.zip]- Use cert-browser.pfx to load into browser to be a NiFi administrator 'DEMO'- Upload other two certificates to Sandbox under '/root/scripts/' and execute below commands, while executing last command enter 'hadoop' as password and 'yes' when asked if

This article delves into the practical aspects of integrating Spark and HBase using Livy, showcasing a comprehensive example that demonstrates the process of reading, processing, and writing data between Spark and HBase. The example utilizes Livy to submit Spark jobs to a YARN cluster, enabling remote execution of Spark applications on HBase data. Prerequisites: Apache Spark installed and configured Apache Livy installed and configured Apache HBase installed and configured HBase Spark Connector jar file available Steps: This step-by-step guide provides a comprehensive overview of how to integrate Spark and HBase using Livy. Step 1: Create an HBase Table Note: If your cluster is kerberized, then you need to provide the proper Ranger HBase permissions to the user and needs to the kinit. Connect to your HBase cluster using the HBase shell:hbase shell​ Create an HBase table named employees with two column families: per and prof:create 'employees', 'per', 'prof'​

Shaun goes into great depth on all the capabilities of CML and DataViz in this blog, using a Python model or script.  Here, I'll write the script in R instead of the sample Python script Shaun shares. There is no need to have an R script with predictive model capabilities.  While an R script with predictive model capabilities is an option, I'll demonstrate deploying a simple R script in CML that adds two columns together. Step 1: Write your R script As explained above, this script adds two columns together (assuming both columns are of type integer).  Within DataViz, we'll pass two columns (of type integer).  The cml_model wrapper is necessary with the PBJ runtime, as explained here. Notice the "json" is what we'll be returning to DataViz, where we expect the "numbers_added" column input[1] and input[2] are representing the two integer columns being passedlibrary(cml) add_numbers <- cml_model(function(args) { json <-'{ "data": { "colnames": [

Introduction This article demonstrates how a Machine Learning(ML) engineer can use the Model Registry Service in Cloudera Machine Learning for cataloging, versioning, and deploying models. The Model Registry can serve as a catalog for models in the DataLake, besides helping provide model lineage information for deployed models for Data and Administrators. For details on how to use the model registry and additional documentation, review the references section below. Creating the Model Registry for the Data Lake Model Registry is one of the core building blocks toward MLOps or Devops for Data Science workflows. It is important to note that a single model registry is created for a CDP DataLake and serves as a model catalog for all the models in the DataLake. To create a model registry, click on CML Control Plane and create a new model registry. If there is an existing model registry for that DataLake / CDP environment, you will not be allowed to create a new one. The screen below shows th

Introduction: This article demonstrates how a Machine Learning(ML) engineer can use the Model Registry Service in Cloudera Machine Learning for cataloging, versioning, and deploying models. ML Engineers work iteratively in an effort to improve model performance using one or more target metrics. This process step is called Experimentation, where the engineer works through a series of trial experiments by changing different parameters of the underlying model, called hyper-parameters, to improve the target metric.  In CML, the Experiments feature allows users to compare experiment runs and visualize the impact of parameters on the target metric (for more information, refer to the article on hyper-parameter tuning here).  Once a chosen experiment run is deemed satisfactory, use the CML Model Registry to register the model. The Model Registry can serve as a centralized catalog for all models that are deemed ready for deployment. Subsequent improvements can then be registered in th

Cloudera Data EngineeringCDE is the Cloudera Data Engineering Service, a containerized managed service for Cloudera Data Platform designed for Large Scale Batch Pipelines with Spark, Airflow and Iceberg. It allows you to submit batch jobs to auto-scaling virtual clusters. As a Cloud-Native service, CDE enables you to spend more time on your applications, and less time on infrastructure.CDE allows you to create, manage, and schedule Apache Spark jobs without the overhead of creating and maintaining Spark clusters. With CDE, you define virtual clusters with a range of CPU and memory resources, and the cluster scales up and down as needed to run your Spark workloads, helping to control your cloud costs.Cloudera Data Engineering provides a command line interface (CLI) client. You can use the CLI to create and update jobs, view job details, manage job resources, run jobs, and more.RequirementsThe following are required in order to reproduce these commands in your CDE environment:A CDE Servi

Cloudera Data EngineeringCDE is the Cloudera Data Engineering Service, a containerized managed service for Cloudera Data Platform designed for Large Scale Batch Pipelines with Spark, Airflow and Iceberg. It allows you to submit batch jobs to auto-scaling virtual clusters. As a Cloud-Native service, CDE enables you to spend more time on your applications, and less time on infrastructure.CDE allows you to create, manage, and schedule Apache Spark jobs without the overhead of creating and maintaining Spark clusters. With CDE, you define virtual clusters with a range of CPU and memory resources, and the cluster scales up and down as needed to run your Spark workloads, helping to control your cloud costs.Cloudera Data Engineering provides a command line interface (CLI) client. You can use the CLI to create and update jobs, view job details, manage job resources, run jobs, and more.CDE Files ResourcesA resource in Cloudera Data Engineering is a named collection of files used by a j

Objective This article provides a quickstart for the SparkXML package in Cloudera Data Engineering. You can run the following commands to parse XML files with Spark in Cloudera Data Engineering.  CDE is the only cloud-native service purpose-built for enterprise data engineering teams. Building on Apache Spark, Data Engineering is an all-inclusive data engineering toolset that enables orchestration automation with Apache Airflow, advanced pipeline monitoring, visual troubleshooting, and comprehensive management tools to streamline ETL processes across enterprise analytics teams. The CDE is fully integrated with the Cloudera Data Platform (CDP), enabling end-to-end visibility and security with SDX as well as seamless integrations with CDP services such as Data Warehouse and Machine Learning. Data Engineering on CDP powers consistent, repeatable, and automated data engineering workflows on a hybrid cloud platform anywhere Spark-XML is a library for parsing and querying XML data with

There is an easy way to check Meta Region Server info from HMaster webUI > System Tables > hbase:meta.  However, anyone can check the same from the command line, or the HMaster UI might not be accessible due to security restrictions. This article explains how to find the Meta Region Server info from the command line. Option 01: The status 'detailed'  shell command can be useful since it has all the hosted region’s info per RS separately along with other useful info.   #hbase shellhbase(main):033:0> status 'detailed'version 2.1.0-cdh6.3.x-SNAPSHOT0 regionsInTransitionactive master: cdh-63x-ie-3.cdh-63x-ie.root.hwx.site:22001 16995050112600 backup mastersmaster coprocessors: [AccessController, MasterAuditCoProcessor]3 live servers cdh-63x-ie-1.cdh-63x-ie.root.hwx.site:22101 1699505010867 requestsPerSecond=0.0, numberOfOnlineRegions=1, usedHeapMB=894, maxHeapMB=31552, numberOfStores=3, numberOfStorefiles=3, storefileUncompressedSizeMB=0, storefileSizeMB=

The Cloudera Data Platform (CDP) is a hybrid data platform designed to deliver faster and easier data management, analytics and AI at enterprise scale. Cloudera AI is one of CDP’s Cloud Native Data Services designed to enable secure, governed Data Science. Data Scientists and Engineers use CML to securely analyze large amounts of data via interactive notebooks. Since 2022 CML enhances this capability with the Data Connections feature by providing boiler template code to access and push down SQL queries to the CDW service. This allows the user to run large scale queries directly in the Data Warehouse while accessing and visualizing results from a simple notebook with a small consumption footprint.In 2023 CML doubled down with CML Custom Data Connections. Custom Data Connections allow the CML user to create their own boiler template code so they or their collaborators can easily connect to 3rd party tools such as Snowflake, Postgres, legacy on-prem databases (Oracle, MSSQL, MySQL),

In the Cloudera, there are two options for fault tolerance in Flink. One is a checkpoint, and the other is a savepoint [1]. Checkpoints are created automatically when enabled and are used for automatically restarting jobs in case of failure. While savepoints are manually triggered, they are always stored externally and are used for starting a "new" job with a previous internal state. Savepoints are what are used when you are performing steps like an upgrade.[2] [1] Checkpoints vs. Savepoints  [2] Savepoint CHECKPOINT ERROR RECOVERY Further, at checkpoints, there are two levels of error recovery in Flink jobs.One is from the yarn side, and the other is from the Flink side. From the yarn side, we have yarn.maximum-failed-containers. This property will check from the Yarn Application master side whether 100 (default) containers have failed or not, and once 100 containers have failed, the Yarn application will fail, and it will not take into consideration any Flink job paramete

OverviewPurpose of this article is to explain a few ways to access metadata of data objects in Cloudera's Hive & Impala data warehouses.Before we get into the specific details, let's understand a typical method to access metadata and data objects in Cloudera Data Platform.Hive Metastore (HMS) database holds the metadata of all Hive & Impala data objects. When user/application is outside the Cloudera cluster, thrift protocol is used to access it.Hive & Impala data warehouses hold all the data objects. When user/application is outside the Cloudera cluster, Cloudera's JDBC/ODBC drivers are used to access it.As you know Cloudera excels at Hybrid architecture and there is a consistent method of accessing metadata and data across different deployment models. Let's now understand how it works.If you're not already familiar with the CDP deployment models, it's highly recommended to check out the following links: Base, Datahub, CDW Data Servic

Dear Team, How to change email id for an account on cloudera community portal.

If you come across this error while doing HBase Service check in a non-kerberized environemnt: "ERROR: Table ambarismoketest does not exist.'"Please follow these steps to resolve it:1. Stop HBase from Ambari. 2. Go to hbase zookeeper CLI and look for $ hbase zkcli$ls /hbase-unsecure/table/ambarismoketest3. If the table exists then delete the table:rmr /hbase-unsecure/table/ambarismoketest4. Start HBase from Ambari5. Now, recreate ambarismoketest table:$ hbase shellhbase> create 'ambarismoketest','family'hbase> quit6. It should fix the issue.

Kudu: Kudu is an open-source distributed storage engine designed for big data [TB and PB]. Kudu is designed for random access and analytical queries on structured data.  It supports scans, random access, and lookups. Although it supports full table scans as an OLAP database and returns results super fast, you should always try to do partition pruning in queries in case you don’t need the entire data set. This will result in fewer scans to the backend server, which in turn will increase the capacity of our Kudu database to handle more queries. Terminology: Kudu backend servers, where data resides on disks, are called Tablet Servers. Kudu tables are partitioned, and each partition is stored in a tablet within the tablet server. Types of Partition in Kudu: Hash Generally a numeric or alphanumeric value Range Multilevel(Hash + Range) To check table partitioning: Open up our SQL Editor and execute. >> Show create table database.tablename. This command will give you table def

It's all about simplicity and cohesion.  With the multiple services within CDP, today I'll focus on finding actively running queries in CDW (Impala) from CML.  With the three steps below, you'll be able to find actively running query progress: Step 1: Find the coordinator URL within CDW (Impala) Within CDW, go to your Virtual Warehouse and select "Edit": Within the Virtual Warehouse, go to the "WEB UI" page: Copy the Coordinator Web UI address, in my example I'll remove the https://"coordinator-web-default-impala.dw-go01-demo-aws.ylcu-atmi.cloudera.site"​ Step 2: Since we're connecting to CDW (Impala) from within CML, I'll set my project's Environment Variables to include my username/password Setting the WORKLOAD_PASSWORD to my workload password Setting the variable WORKLOAD_USER to my username Step 3: Within my CML Notebook (in my case PBJ), I'll copy the following code, replacing the coordinator with the coordinator from Step 1:   import osimport requestsimport pan

Kudu Command Line Copy: The Copy Table command can be used to copy one Kudu table to another. The two tables could be in the same cluster or not. The two tables must have the same table schema but could have different partition schemas. Alternatively, the tool can create the new table using the same table and partition schema as the source table. Example: Full table copy command: kudu table copy <master_addresses> <source_table_name> <dest_master_addresses> -dst_table=<table_name> -write_type=upsert -num_threads=3 Incremental copy command: kudu table copy <master_addresses> <source_table_name> <dest_master_addresses> -dst_table=<table_name> -write_type=upsert -num_threads=3 -predicates='["AND", [">=", "some_value", 234]]' Spark Backup Utility: Kudu supports both full and incremental table backups via a job implemented using Apache Spark. Additionally, it supports restoring tables from full and incremental backups via a rest