Community Articles

Introduction:In large Hadoop clusters, efficiently managing block replication and decommissioning of DataNodes is crucial for maintaining system performance and reliability. However, updating Namenode configuration parameters to optimize these processes often requires a Namenode restart, causing downtime and potential disruptions to cluster operations. In this article, we'll explore a procedure to expedite block replication and DataNode decommissioning in HDFS without the need for a Namenode restart. Procedure:Identify Namenode Process Directory:Locate the Namenode process directory for the current active Namenode. This directory typically resides in /var/run/cloudera-scm-agent/process/ followed by a folder that looks like "###-hdfs-NAMENODE"Modify Configuration Parameters:Edit the hdfs-site.xml file in the Namenode process directory.Adjust the following parameters to the recommended values:dfs.namenode.replication.max-streams: Increase to a recommended value (e.g., 100).dfs.namen

I've run into this issue a number of time over the years. It's typically associated with environments using Active Directory or FreeIPA for Kerberos authentication. The tell-tale of this problem is this: even though an interactive kinit (using a password) works for a user, she/he cannot authenticate with a keytab, getting the error: "kinit: Preauthentication failed while getting initial credentials". The problem can affect some users but not others when using Active Directory. NOTE: The problem described below is not the root cause of all the "kinit: Preauthentication failed while getting initial credentials" errors. In these cases, the keytab is typically created with the ktutil utility, using the AES encryption type. The following ktutil commands are typically used to generate the keytab: addent -password -p <username> -k 0 -e aes256-ctswrite_kt <username>.keytab  When the keytab is created, a "salt" is used to encrypt the contents of the keytab.

Cloudera documentation is now available at https://docs.cloudera.com/:The CDP documentation is divided in the following sections corresponding to CDP services and components:Management Console Workload ManagerData CatalogReplication ManagerData HubData WarehouseMachine LearningCloudera RuntimeCloudera ManagerEach of these documentation sections includes its own Release Notes document. Furthermore, when using CDP, you can access contextual help by clicking on the Help icon in the bottom left corner:

Purpose: Access CDW Iceberg table with Snowflake Architecture Image source: Configure an external volume for Iceberg tables Initiate Configure an External Volume in Snowflake Step 1: Configure access permissions for the S3 bucket Create a policy in AWS IAM. I created a policy called zzeng-Snowflake-ext:{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:GetObjectVersion", "s3:DeleteObject", "s3:DeleteObjectVersion" ], "Resource": "arn:aws:s3:::<my-bucket>/data/zzeng/*" }, { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetBucketLocation" ], "Resource": "arn:aws:s3:::<my-bucket>", "Condition": { "StringLike": { "s3:prefix": [

Goal : Use CLI to create / import a dataflow in CDF-PC Then, run Teraform to create/import Dataflow into CDF-PC. Process :  Install CDP CLI (I am using MacOS, so I install cdpcli in this way. pip3 install cdpcliThen confirm it's installed correctly[zzeng@zeng-mbp ~]$ ll ~/Library/Python/3.9/bin/ | grep cdp-rwxr-xr-x@ 1 zzeng staff 250 Mar 4 12:22 cdp-rwxr-xr-x@ 1 zzeng staff 250 Mar 4 12:22 cdp_completer[zzeng@zeng-mbp ~]$ export PATH="$HOME/Library/Python/3.9/bin:$PATH"[zzeng@zeng-mbp ~]$ cdp --version0.9.107[zzeng@zeng-mbp ~]$ Configure CLI: (Reference Link) from [Management Console] ->[User] -> [Profile] Then I can find your API key here. If it does not exist, create one. Then, configure the API key in my MacOS (Reference):cdp configure After this config,  check the latest status:[zzeng@zeng-mbp ~]$ cdp iam get-user{ "user": { "userId": ***** "status": "ACTIVE", "workloadPasswordDetails": { "isPasswordSet": t

Cloudera Data Platform provides two complementing services for data processing. Cloudera Data Engineering for preparing and organising data to be consumed by Data Scientists in Cloudera AI. This article provides a high level example of how to call CDE from CML in Python. ## Part 0: Imports# Requests will be used to make REST requests to the CDE service# JSON is used to manage payloads and responses# REST calls are made over HTTP and use BasicAuth# cde_endpoint is set to the high level URI to the CDE cluster endpointimport requestsimport jsonfrom requests.auth import HTTPBasicAuthcde_endpoint='cde-<cluster>.cloudera.site' The first step is establishing the credentials required to call the API. This is done by calling the Knox proxy and requesting a token. The token is then passed using the "Authorization" header. ## Part 1: Connect to Knox service.# Retrieve the JWT token and parse it for the 'access_token' part.# Need to pass workload username and password which is set as a proj

ObjectiveCDE is the Cloudera Data Engineering Service, a containerized managed service for Cloudera Data Platform designed for Large Scale Batch and Streaming Pipelines with Spark, Airflow and Iceberg. It allows you to submit batch jobs to auto-scaling virtual clusters. As a Cloud-Native service, CDE enables you to spend more time on your applications, and less time on infrastructure.CDE allows you to create, manage, and schedule Apache Spark jobs without the overhead of creating and maintaining Spark clusters. With CDE, you define virtual clusters with a range of CPU and memory resources, and the cluster scales up and down as needed to run your Spark workloads, helping to control your cloud costs.CDE Sessions are short-lived development environment used by Data Engineers and Analysts in order to iterate upon and build Spark workloads, prototype pipelines, or simply explore data. You can use CDE Sessions in CDE Virtual Clusters of type "All Purpose - Tier 2".In this example we will use

Spark Scala Version Compatibility Matrix 1. Introduction Apache Spark being a widely used framework for big data processing, relies heavily on Scala as its primary programming language. Ensuring compatibility between different versions of Spark and Scala is essential for developers to leverage the latest features and optimizations while maintaining stability in their Spark applications.  In this article, we'll provide a comprehensive overview of the compatibility matrix between different versions of Spark and Scala, helping developers choose the right combination for their projects. Key Considerations: Several factors influence compatibility between Spark and Scala versions: API changes: New features or modifications in Spark APIs might require specific Scala versions for proper compilation and execution. Library dependencies: Third-party libraries used within your Spark application might have compatibility requirements with both Spark and Scala versions. Community support:

Spark and Java versions Supportability Matrix 1. Introduction: Apache Spark is a powerful open-source distributed computing system widely used for big data processing and analytics. However, choosing the right Java version for your Spark application is crucial for optimal performance, security, and compatibility.  This article dives deep into the officially supported Java versions for Spark, along with helpful advice on choosing the right one for your project. 2. Matrix Table Spark Version Supported Java Version(s) Java 8 Java 11 Java 17 Java 21 Deprecated Java Version(s) 3.5.1 Java 8*/11/17 Yes Yes Yes No Java 8 prior to version 8u371 support is deprecated 3.5.0 Java 8*/11/17 Yes Yes Yes No Java 8 prior to version 8u371 support is deprecated 3.4.2 Java 8*/11/17 Yes Yes Yes No Java 8 prior to version 8u362 support is deprecated 3.4.1 Java 8*/11/17 Yes Yes Yes No Java 8 prior to version 8u362 support is

What is DataFlint DataFlint is an open-source D-APM (Data-Application Performance Monitoring) for Apache Spark, built for big data engineers. For more information see dataflint.gitbook.io/dataflint-for-spark or the git-repo itself  How to Integrate DataFlint in CDP DataFlint supports spark 3.2+, so you will need a spark3 parcel  For spark-submit see DataFlint:    spark-submit--packages io.dataflint:spark_2.12:0.3.1 \--conf spark.plugins=io.dataflint.spark.SparkDataflintPlugin \...   To Install on Spark History Server: download DataFlint jar for scala2.12  Copy it to all servers that have an instance of spark-history Role to spark 3 parcel dir:/opt/cloudera/parcels/SPARK3/lib/spark3/jars​Beware /opt/cloudera/parcels/SPARK3 is a symlink to the current SPARK3 active parcel and is controlled by Cloudera Manager. Go to CM >cluster >Spark 3 >Configuration >spark3-conf/spark-history-server.conf_role_safety_valv

This article describes various "recipes" on how to accomplish certain tasks with the NiFi processor ExecuteScript, with examples given in Groovy, Jython, Javascript (Nashorn), and JRuby. Recipes in this article series include: Part 1 - Introduction to the NiFi API and FlowFiles Getting a flow file from an incoming queue Creating new flow files Working with flow file attributes Transferring a flow file Logging Part 2 - FlowFile I/O and Error Handling Reading from a flow file Writing to a flow file Reading and writing to/from a flow file Error Handling Part 3 - Advanced Features Using Dynamic Properties Adding Modules State Management Accessing Controller Services Introduction ExecuteScript is a versatile processor that allows the user to code custom logic in a programming language that will be executed each time the ExecuteScript processor is triggered. The following variable bindings are provided to the script to enable access to NiFi components: session: This is a reference to

SummaryIn this article, we learn how to use CML to automate Machine Learning workflows between development and production environments using an automation toolchain such as Gitlab. This solution can be extended to other platforms such as Github( using Github actions) or Atlassian (using Bitbucket pipelines).Some prior understanding of how the DevOps pipeline works and how it can be applied in Machine learning is assumed here. There are some good resources online that can be referred for a deep-dive, but just enough information is provided below. A (very) brief Introduction to CI-CD for Machine LearningA DevOps pipeline is a set of automated processes and tools that allows software development teams to build and deploy code to a production environment. A few of the important components of the DevOps pipeline include Continuous Integration (CI) and Continuous Delivery and Deployment (CD), where:  Continuous Integration is the practice or ability to make frequent commi

LIMITATIONS OF TEMPLATES If a user wants to change the flows using templates, i.e., downloading templates and uploading them back, then the following issues will occur:  The template will not preserve the state. It is not supposed to. The template is just a blueprint of the components, not their data. New controller services will be created instead of using already existing ones, which is expected behavior in templates. Templates will be unsupported in CFM 4.0/ NiFi 2.0 and there are not active improvements being done to them. FLOW DEFINITION The better solution in this case will be to use flow definitions instead of templates. The steps to use flow definitions are:  Right-click on Process Group > Click on Download Flow Definition > select the option to download flow JSON with or without external controller services. “External” here means services referenced inside the PG, but existing at a higher level. Then, if you import that flow definition (JSON) that has refer

NiFi JSON to DDL Custom ProcessorJava ClassJUnitThis is further enhanced version of the idea started here: https://community.hortonworks.com/articles/154957/converting-json-to-sql-ddl.html?es_p=6294995There was some discussion on linkedin about the previous article being a good processor, so I decided to do that. This is pretty basic, but it handles most types okay. Date and number processing is a bit hacky, but guesses some types.To install, copy the NAR file that you build or download from Github to your NiFi/lib directories and restart those servers.Add the New Processor to Your FlowConfigure the Processor with a table type (that is ignored in this version)Configure the Processor with a table name (this is important)JsonToDDLProcessor Generated DocsI configured my table name to be the filename without an extension for JSONOutput in NiFiExample FlowEnhancements In Consideration: Apache OpenNLPApache TikaAttribute Cleaner EnhancementDeep Learning for Determining TypesMachine Lea

Upgrade your COD cluster without downtime Released on August 10, 2023 You can now upgrade the Cloudera Runtime version and the operating system version of the database that uses HDFS and Object Store without Ephemeral Storage, in a rolling restart mode. Previously, while performing a Cloudera Runtime upgrade, users experienced HBase or Phoenix service downtime provided by COD. You can now avoid a service outage of a COD cluster while performing a Cloudera Runtime upgrade. This feature ensures continuous service availability. To upgrade the Cloudera Runtime version, use the --rolling-upgrade | --no-rolling-upgrade option on CDP CLI with the upgrade-database command. cdp opdb upgrade-database --environment <environment-name> --database <database-name> --runtime <runtime-version> [--rolling-upgrade | --no-rolling-upgrade] To upgrade the operating system version, use the --os-upgrade-only option along with the --rolling-upgrade option in the upgrade-database command. cdp

Short Description:This Tutorial describes how to add fields,removing not required fields and change values of fields in flowfile.ArticleIntroductionUsing UpdateRecord processor we are can update the contents of flowfile.Overview of article:Below sections describes the changes that are going to happen to the input flowfile content vs output flowfile contents.Input:input Json record as follows and having 6 fields/elements in it.[ { "id" : 1, "name" : "foo", "age" : 20, "state" : "FLORIDA", "ts_milli" : 1525526792098, "ts" : "2018-05-03 10:10:10.123"}]Expected Output:I'm writing output in Avro Format but for viewing purpose i have converted the output flowfile to json.[ { "id" : "1", "rename_id" : "1", //newly added field based on id "state" : "florida", //changed the value of field "rename_state" : "FLORIDA", //newly added field based on state "unique_id" : "1-416425265990923-FLORIDA-aab46988-6a27-4008-9a4d-65655abe9c3c",//id-filename-state-UUID "ts_milli" : 1525526792098,

Hive 3.0 introduced an option to re-attempt a failed Hive query, in case the first run fails. It would only make sense if we fixed whatever was the issue in the previous run. We'll discuss the ways to configure this once without having to intervene after each failure event.The following Hive property enables query re-execution. This should be enabled out of the box.hive.query.reexecution.enabled=true;hive.query.reexecution.strategies=overlay,reoptimize,recompile_without_cbo,reexecute_lost_am;Re-execution strategies:OverlayUsing this method, we can set a Hive property that should be applied on the re-execution. It works by adding a configuration subtree as an overlay to the actual hive settings(reexec.overlay.*).set reexec.overlay.{hive_property}=new_valueEvery hive setting which has a prefix of "reexec.overlay" will be set for all re-executions.e.g.In case our Hive queries fail with OOM while performing Map Joins, which could occur when we do not have correct stats for the tables, we c

Objective: Describe a process to perform DI check using spark JDBC and also oozie spark action.Most of the ETL pipelines involve DI checks between source system and ETL environment to ensure that the record counts match, and also sum/average of certain significant columns is compared on both source & ETL environments. We will see how DI check can be performed and results are stored for audit purpose using spark JDBC. We will also talk about how to prevent storing clear text password in order to make JDBC call using spark.The following components need to be written in order to accomplish this. PySpark code that invokes the Java class and retrieves clear text password at runtime & does DI check.Java class to decrypt the password using hadoop credentials java APIOozie spark action to trigger the spark code that performs DI check (counts)PySpark Code for DI Check: This python module calls a java method to decrypt the password and uses it in the spark jdbc call to get the row count

CDE is the Cloudera Data Engineering Service, a containerized managed service for the Cloudera Data Platform designed for Large Scale Batch Pipelines with Spark, Airflow, and Iceberg. It allows you to submit batch jobs to auto-scaling virtual clusters. As a Cloud-Native service, CDE enables you to spend more time on your applications and less time on infrastructure. CDE allows you to create, manage, and schedule Apache Spark jobs without the overhead of creating and maintaining Spark clusters. With CDE, you define virtual clusters with a range of CPU and memory resources, and the cluster scales up and down as needed to run your Spark workloads, helping to control your cloud costs. Cloudera Data Engineering provides a command-line interface (CLI) client. The CDE CLI doesn't just allow you to create CDE Spark Jobs with syntax that is nearly identical to that of a Spark Submit. The CDE CLI also offers you invaluable advantages as part of your daily Spark Pipeline development and managemen

High level Problem ExplanationThe main issue impacting Namenode performance is the global FSNamesystem lock during read and write operations. Handling tasks in larger directories, especially those involving millions of files, can lead to high lock times and overall slowness in operations like extensive file listings, retrieving block locations, handling large deletions, performing DU (Usage) and BDR snapshot activities. ------------ (Global lock - NameSystem) -------- | | | Directory set (InodeMap) Block set(BlockMap) Cluster information(DatanodeManager)Detailed Technical ExplanationRPC request #getListing:DFSClient initiates getList for the given path, invoking DistributedFileSystem#listStatus. It is important to note that this process isn't atomic, and the directory entries may be retrieved from the NameNode multiple times.DFSClient retrieves the initial batch of entr

NiFi User Authentication with LDAP in HW SandboxPrerequisite 1) Assuming you already have NiFi-0.4.0/HDF-1.1 downloaded on your HW Sandbox, else execute below after ssh connectivity to sandbox is established.cd /opt/wget http://public-repo-1.hortonworks.com/HDF/1.1.0.0/nifi-1.1.0.0-10-bin.zipunzip nifi-1.1.0.0-10-bin.zip2) Make sure Knox is installed on your sandbox and demo LDAP is started via AmbariSteps:1. I already created Certification Authorities and client certificates at www.tinycert.orgHow to create certificates as well configurations can be found in below article:https://community.hortonworks.com/content/kbentry/886/securing-nifi-step-by-step.htmlIf you are too lazy to create them, try with mine 🙂 [Attached as certificates.zip]- Use cert-browser.pfx to load into browser to be a NiFi administrator 'DEMO'- Upload other two certificates to Sandbox under '/root/scripts/' and execute below commands, while executing last command enter 'hadoop' as password and 'yes' when asked if

This article delves into the practical aspects of integrating Spark and HBase using Livy, showcasing a comprehensive example that demonstrates the process of reading, processing, and writing data between Spark and HBase. The example utilizes Livy to submit Spark jobs to a YARN cluster, enabling remote execution of Spark applications on HBase data. Prerequisites: Apache Spark installed and configured Apache Livy installed and configured Apache HBase installed and configured HBase Spark Connector jar file available Steps: This step-by-step guide provides a comprehensive overview of how to integrate Spark and HBase using Livy. Step 1: Create an HBase Table Note: If your cluster is kerberized, then you need to provide the proper Ranger HBase permissions to the user and needs to the kinit. Connect to your HBase cluster using the HBase shell:hbase shell​ Create an HBase table named employees with two column families: per and prof:create 'employees', 'per', 'prof'​

Shaun goes into great depth on all the capabilities of CML and DataViz in this blog, using a Python model or script.  Here, I'll write the script in R instead of the sample Python script Shaun shares. There is no need to have an R script with predictive model capabilities.  While an R script with predictive model capabilities is an option, I'll demonstrate deploying a simple R script in CML that adds two columns together. Step 1: Write your R script As explained above, this script adds two columns together (assuming both columns are of type integer).  Within DataViz, we'll pass two columns (of type integer).  The cml_model wrapper is necessary with the PBJ runtime, as explained here. Notice the "json" is what we'll be returning to DataViz, where we expect the "numbers_added" column input[1] and input[2] are representing the two integer columns being passedlibrary(cml) add_numbers <- cml_model(function(args) { json <-'{ "data": { "colnames": [

Introduction This article demonstrates how a Machine Learning(ML) engineer can use the Model Registry Service in Cloudera Machine Learning for cataloging, versioning, and deploying models. The Model Registry can serve as a catalog for models in the DataLake, besides helping provide model lineage information for deployed models for Data and Administrators. For details on how to use the model registry and additional documentation, review the references section below. Creating the Model Registry for the Data Lake Model Registry is one of the core building blocks toward MLOps or Devops for Data Science workflows. It is important to note that a single model registry is created for a CDP DataLake and serves as a model catalog for all the models in the DataLake. To create a model registry, click on CML Control Plane and create a new model registry. If there is an existing model registry for that DataLake / CDP environment, you will not be allowed to create a new one. The screen below shows th

Introduction: This article demonstrates how a Machine Learning(ML) engineer can use the Model Registry Service in Cloudera Machine Learning for cataloging, versioning, and deploying models. ML Engineers work iteratively in an effort to improve model performance using one or more target metrics. This process step is called Experimentation, where the engineer works through a series of trial experiments by changing different parameters of the underlying model, called hyper-parameters, to improve the target metric.  In CML, the Experiments feature allows users to compare experiment runs and visualize the impact of parameters on the target metric (for more information, refer to the article on hyper-parameter tuning here).  Once a chosen experiment run is deemed satisfactory, use the CML Model Registry to register the model. The Model Registry can serve as a centralized catalog for all models that are deemed ready for deployment. Subsequent improvements can then be registered in th