Community Articles

Cloudera Data EngineeringCDE is the Cloudera Data Engineering Service, a containerized managed service for Cloudera Data Platform designed for Large Scale Batch Pipelines with Spark, Airflow and Iceberg. It allows you to submit batch jobs to auto-scaling virtual clusters. As a Cloud-Native service, CDE enables you to spend more time on your applications, and less time on infrastructure.CDE allows you to create, manage, and schedule Apache Spark jobs without the overhead of creating and maintaining Spark clusters. With CDE, you define virtual clusters with a range of CPU and memory resources, and the cluster scales up and down as needed to run your Spark workloads, helping to control your cloud costs.Cloudera Data Engineering provides a command line interface (CLI) client. You can use the CLI to create and update jobs, view job details, manage job resources, run jobs, and more.RequirementsThe following are required in order to reproduce these commands in your CDE environment:A CDE Servi

Cloudera Data EngineeringCDE is the Cloudera Data Engineering Service, a containerized managed service for Cloudera Data Platform designed for Large Scale Batch Pipelines with Spark, Airflow and Iceberg. It allows you to submit batch jobs to auto-scaling virtual clusters. As a Cloud-Native service, CDE enables you to spend more time on your applications, and less time on infrastructure.CDE allows you to create, manage, and schedule Apache Spark jobs without the overhead of creating and maintaining Spark clusters. With CDE, you define virtual clusters with a range of CPU and memory resources, and the cluster scales up and down as needed to run your Spark workloads, helping to control your cloud costs.Cloudera Data Engineering provides a command line interface (CLI) client. You can use the CLI to create and update jobs, view job details, manage job resources, run jobs, and more.CDE Files ResourcesA resource in Cloudera Data Engineering is a named collection of files used by a j

Objective This article provides a quickstart for the SparkXML package in Cloudera Data Engineering. You can run the following commands to parse XML files with Spark in Cloudera Data Engineering.  CDE is the only cloud-native service purpose-built for enterprise data engineering teams. Building on Apache Spark, Data Engineering is an all-inclusive data engineering toolset that enables orchestration automation with Apache Airflow, advanced pipeline monitoring, visual troubleshooting, and comprehensive management tools to streamline ETL processes across enterprise analytics teams. The CDE is fully integrated with the Cloudera Data Platform (CDP), enabling end-to-end visibility and security with SDX as well as seamless integrations with CDP services such as Data Warehouse and Machine Learning. Data Engineering on CDP powers consistent, repeatable, and automated data engineering workflows on a hybrid cloud platform anywhere Spark-XML is a library for parsing and querying XML data with

There is an easy way to check Meta Region Server info from HMaster webUI > System Tables > hbase:meta.  However, anyone can check the same from the command line, or the HMaster UI might not be accessible due to security restrictions. This article explains how to find the Meta Region Server info from the command line. Option 01: The status 'detailed'  shell command can be useful since it has all the hosted region’s info per RS separately along with other useful info.   #hbase shellhbase(main):033:0> status 'detailed'version 2.1.0-cdh6.3.x-SNAPSHOT0 regionsInTransitionactive master: cdh-63x-ie-3.cdh-63x-ie.root.hwx.site:22001 16995050112600 backup mastersmaster coprocessors: [AccessController, MasterAuditCoProcessor]3 live servers cdh-63x-ie-1.cdh-63x-ie.root.hwx.site:22101 1699505010867 requestsPerSecond=0.0, numberOfOnlineRegions=1, usedHeapMB=894, maxHeapMB=31552, numberOfStores=3, numberOfStorefiles=3, storefileUncompressedSizeMB=0, storefileSizeMB=

The Cloudera Data Platform (CDP) is a hybrid data platform designed to deliver faster and easier data management, analytics and AI at enterprise scale. Cloudera AI is one of CDP’s Cloud Native Data Services designed to enable secure, governed Data Science. Data Scientists and Engineers use CML to securely analyze large amounts of data via interactive notebooks. Since 2022 CML enhances this capability with the Data Connections feature by providing boiler template code to access and push down SQL queries to the CDW service. This allows the user to run large scale queries directly in the Data Warehouse while accessing and visualizing results from a simple notebook with a small consumption footprint.In 2023 CML doubled down with CML Custom Data Connections. Custom Data Connections allow the CML user to create their own boiler template code so they or their collaborators can easily connect to 3rd party tools such as Snowflake, Postgres, legacy on-prem databases (Oracle, MSSQL, MySQL),

In the Cloudera, there are two options for fault tolerance in Flink. One is a checkpoint, and the other is a savepoint [1]. Checkpoints are created automatically when enabled and are used for automatically restarting jobs in case of failure. While savepoints are manually triggered, they are always stored externally and are used for starting a "new" job with a previous internal state. Savepoints are what are used when you are performing steps like an upgrade.[2] [1] Checkpoints vs. Savepoints  [2] Savepoint CHECKPOINT ERROR RECOVERY Further, at checkpoints, there are two levels of error recovery in Flink jobs.One is from the yarn side, and the other is from the Flink side. From the yarn side, we have yarn.maximum-failed-containers. This property will check from the Yarn Application master side whether 100 (default) containers have failed or not, and once 100 containers have failed, the Yarn application will fail, and it will not take into consideration any Flink job paramete

OverviewPurpose of this article is to explain a few ways to access metadata of data objects in Cloudera's Hive & Impala data warehouses.Before we get into the specific details, let's understand a typical method to access metadata and data objects in Cloudera Data Platform.Hive Metastore (HMS) database holds the metadata of all Hive & Impala data objects. When user/application is outside the Cloudera cluster, thrift protocol is used to access it.Hive & Impala data warehouses hold all the data objects. When user/application is outside the Cloudera cluster, Cloudera's JDBC/ODBC drivers are used to access it.As you know Cloudera excels at Hybrid architecture and there is a consistent method of accessing metadata and data across different deployment models. Let's now understand how it works.If you're not already familiar with the CDP deployment models, it's highly recommended to check out the following links: Base, Datahub, CDW Data Servic

If you come across this error while doing HBase Service check in a non-kerberized environemnt: "ERROR: Table ambarismoketest does not exist.'"Please follow these steps to resolve it:1. Stop HBase from Ambari. 2. Go to hbase zookeeper CLI and look for $ hbase zkcli$ls /hbase-unsecure/table/ambarismoketest3. If the table exists then delete the table:rmr /hbase-unsecure/table/ambarismoketest4. Start HBase from Ambari5. Now, recreate ambarismoketest table:$ hbase shellhbase> create 'ambarismoketest','family'hbase> quit6. It should fix the issue.

Kudu: Kudu is an open-source distributed storage engine designed for big data [TB and PB]. Kudu is designed for random access and analytical queries on structured data.  It supports scans, random access, and lookups. Although it supports full table scans as an OLAP database and returns results super fast, you should always try to do partition pruning in queries in case you don’t need the entire data set. This will result in fewer scans to the backend server, which in turn will increase the capacity of our Kudu database to handle more queries. Terminology: Kudu backend servers, where data resides on disks, are called Tablet Servers. Kudu tables are partitioned, and each partition is stored in a tablet within the tablet server. Types of Partition in Kudu: Hash Generally a numeric or alphanumeric value Range Multilevel(Hash + Range) To check table partitioning: Open up our SQL Editor and execute. >> Show create table database.tablename. This command will give you table def

It's all about simplicity and cohesion.  With the multiple services within CDP, today I'll focus on finding actively running queries in CDW (Impala) from CML.  With the three steps below, you'll be able to find actively running query progress: Step 1: Find the coordinator URL within CDW (Impala) Within CDW, go to your Virtual Warehouse and select "Edit": Within the Virtual Warehouse, go to the "WEB UI" page: Copy the Coordinator Web UI address, in my example I'll remove the https://"coordinator-web-default-impala.dw-go01-demo-aws.ylcu-atmi.cloudera.site"​ Step 2: Since we're connecting to CDW (Impala) from within CML, I'll set my project's Environment Variables to include my username/password Setting the WORKLOAD_PASSWORD to my workload password Setting the variable WORKLOAD_USER to my username Step 3: Within my CML Notebook (in my case PBJ), I'll copy the following code, replacing the coordinator with the coordinator from Step 1:   import osimport requestsimport pan

Kudu Command Line Copy: The Copy Table command can be used to copy one Kudu table to another. The two tables could be in the same cluster or not. The two tables must have the same table schema but could have different partition schemas. Alternatively, the tool can create the new table using the same table and partition schema as the source table. Example: Full table copy command: kudu table copy <master_addresses> <source_table_name> <dest_master_addresses> -dst_table=<table_name> -write_type=upsert -num_threads=3 Incremental copy command: kudu table copy <master_addresses> <source_table_name> <dest_master_addresses> -dst_table=<table_name> -write_type=upsert -num_threads=3 -predicates='["AND", [">=", "some_value", 234]]' Spark Backup Utility: Kudu supports both full and incremental table backups via a job implemented using Apache Spark. Additionally, it supports restoring tables from full and incremental backups via a rest

Objective Great Expectations is a Python-based open-source library for validating, documenting, and profiling your data. It helps you maintain data quality and improve communication about data between teams. Software developers have long known that automated testing is essential for managing complex codebases. Great Expectations brings the same discipline, confidence, and acceleration to data science and data engineering teams. (Source: Great Expectations) CDP Data Engineering is the only cloud-native service purpose-built for enterprise data engineering teams. Building on Apache Spark, Data Engineering is an all-inclusive data engineering toolset that enables orchestration automation with Apache Airflow, advanced pipeline monitoring, visual troubleshooting, and comprehensive management tools to streamline ETL processes across enterprise analytics teams. Data Engineering is fully integrated with the Cloudera Data Platform, enabling end-to-end visibility and security with SDX as we

ObjectiveCDP Data Engineering (CDE) is the only cloud-native service purpose-built for enterprise data engineering teams. Building on Apache Spark, Data Engineering is an all-inclusive data engineering toolset that enables orchestration automation with Apache Airflow, advanced pipeline monitoring, visual troubleshooting, and comprehensive management tools to streamline ETL processes across enterprise analytics teams.Apache Sedona™ (formerly known as "GeoSpark") is a cluster computing system for processing large-scale spatial data. Sedona extends existing cluster computing systems, such as Apache Spark and Apache Flink, with a set of out-of-the-box distributed Spatial Datasets and Spatial SQL that efficiently load, process, and analyze large-scale spatial data across machines.Sedona jobs can run in CDE with minor configuration changes. This article will present a simple workflow to help you get started with Spark Geospatial use cases in CDE.All code, dependencies and CLI commands a

What is Cloudera Data EngineeringCDP Data Engineering (CDE) is the only cloud-native service purpose-built for enterprise data engineering teams. Building on Apache Spark, Data Engineering is an all-inclusive data engineering toolset that enables orchestration automation with Apache Airflow, advanced pipeline monitoring, visual troubleshooting, and comprehensive management tools to streamline ETL processes across enterprise analytics teams.Typically CDE users leverage the CLI and API to accomplish tasks such as creating, running and orchestrating Spark and Airflow Jobs, uploading files such as Python wheel files, Jars, etc to the Virtual Cluster, and creating reusable Python environments for Spark and Airflow Jobs. CDEPY is a package that allows you to do all the above with the convenience of Python. With it you can remotely connect to a Virtual Cluster from your local machine or 3rd party tool as long as it supports Python. It is available on PyPi at this URL and can be easi

If you need to connect HiveServer2 from Third parties like Dbbeaver OR PowerBI or any Java client, you can connect using the Cloudera JDBC driver.The driver can be downloaded from Hive JDBC Connector 2.6.21 for Cloudera Enterprise. We will cover the scenario for SSL, Zookeeper,Kerberos, LDAP, and LoadBalancer to connect to HiveServer2 using the driver.    Please note that the Beeline URL is not the same as the JDBC driver. The list of properties for the JDBC driver is listed at Cloudera JDBC Driver 2.6.21 for Apache Hive.You need to use com.cloudera.hive.jdbc.HS2Driver class to connect to HiveServer2.Lets assume:  Host: c2345.node.cloudera.com  Kerberos Realm: EXAMPLE.COMPlain Connection:   jdbc:hive2://c2345.node.cloudera.com:10000/default;LogLevel=6;LogPath=/tmp    where LogLevel =6 ==> is more verbose level of loggingKerberos+SSL+binary:  jdbc:hive2://c2345.node.cloudera.com:10000/default;SS

Apache Spark is a unified analytics engine for large-scale data processing. It provides high-level APIs in Java, Scala, Python, and R, and an optimized engine that supports general execution graphs.  The below table provides you links to features released in each Spark 3 minor version: Spark Version Documentation Link Release Date Spark 3.0.0 https://spark.apache.org/docs/3.0.0/ 2020-06-16 Spark 3.0.1 https://spark.apache.org/docs/3.0.1/ 2020-11-05 Spark 3.0.2 https://spark.apache.org/docs/3.0.2/ 2021-02-19 Spark 3.0.3 https://spark.apache.org/docs/3.0.3/ 2022-06-17 Spark 3.1.1 https://spark.apache.org/docs/3.1.1/ 2021-03-02 Spark 3.1.2 https://spark.apache.org/docs/3.1.2/ 2022-06-17 Spark 3.1.3 https://spark.apache.org/docs/3.1.3/ 2022-06-17 Spark 3.2.0 https://spark.apache.org/docs/3.2.0/ 2021-10-13 Spark 3.2.1 https://spark.apache.org/docs/3.2.1/ 2022-06-17 Spark 3.2.2 https://spark.apache.org/docs/3.2.2/ 2022-07-15 Spark 3.2.3 https://spark.apache.org/

Install Install Rust from Install Rust This is interactive so needs to run in a terminalcurl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | shHello WorldRust Hello World https://doc.rust-lang.org/book/ch01-02-hello-world.html​ Reload your session and terminal to update the PATH env Runrustc main.rsmain.rs…​ Why Rust?  Original Post from Stack Overflow:  what is rust and why is it so popular Rust has been Stack Overflow’s most loved language for four years in a row, indicating that many of those who have had the opportunity to use Rust have fallen in love with it. However, the roughly 97% of survey respondents who haven’t used Rust may wonder, “What’s the deal with Rust?” Rust’s static typing does its best to get out of the programmer’s way while encouraging long-term maintainability. Rust gives you the choice of storing data on the stack or the heap and determines at compile time when memory is no longer needed and can be cleaned up. The biggest benefit Rust c

Introduction The new method of capturing Cloudera Manager performance profile utilizes the Java Flight Recorder (JFR) feature of JVM. (Running Java Flight Recorder)  JFR works on OpenJDK 11 and higher,  also OpenJDK 8u262 and higher.  For older JDK8, use the older method for capturing performance profiles. (Steps to collect Cloudera Manager profiling data to troubleshoot performance issues - you need subscription to access this article) The performance capture files can be analyzed with the JMC tool - JDK Mission Control (JMC)  Pre-requisites Ensure recent JDK # /usr/java/default/bin/java -versionopenjdk version "1.8.0_232" OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_232-b09) OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.232-b09, mixed mode) JDK 8u232 is not recent enough to support JFR.  Will not work.  Must be OpenJDK 8u262 or higher. —--- # /usr/java/default/bin/java -versionopenjdk version "11.0.3" 2019-04-16 LTS OpenJDK Runt

Hey Cloudera Community!We're thrilled to announce our latest addition to the Cloudera AMP catalog - Text Summarization and more with Amazon Bedrock! If you've been keen on tapping into the power of foundation models from leading AI companies like Amazon, this AMP is crafted just for you.What are AMPs?Applied ML Prototypes (AMPs) are pre-packaged ML projects designed for easy deployment within Cloudera AI. AMPs enable data scientists to go from an idea to a fully working ML use case in a fraction of the time.What's Inside:Guided Setup: Navigate effortlessly through the setup process and start using the Amazon Bedrock API in no time.Highlighted Models: Dive deep with two powerful models:amazon.titan-tg1-largeanthropic.claude-v2 (Supports a whopping 100K tokens in the prompt!)Flexibility in Action: Customize the instruction prompts as needed to instruct the models for diverse text generation tasks.API Guidance: Benefit from clear directions on prompt formats and request API schema for has

Synopsis.ORC is a columnar storage format for Hive.This document is to explain how creation of ORC data files can improve read/scan performance when querying the data. TEZ execution engine provides different ways to optimize the query, but it will do the best with correctly created ORC files.ORC Creation Strategy.Example:CREATE [EXTERNAL] TABLE OrcExampleTable(clientid int, name string, address string, age int)stored as orcTBLPROPERTIES ("orc.compress"="ZLIB","orc.compress.size"="262144","orc.create.index"="true","orc.stripe.size"="268435456","orc.row.index.stride"="3000","orc.bloom.filter.columns"="clientid,age,name");Ingesting data into Hive tables heavily depends on usage patterns. In order to make queries running efficiently, ORC files should be created to support those patterns. -Identify most important/frequent queries that will be running against your data set (based on filter or JOIN conditions)-Configure optimal data file size-Configure stripe and stride size-Distribute and so

Pandas is a fast, powerful and flexible Python data analysis and manipulation tool. It has gained incredible popularity in the Data Science community as one of the most important  packages for data wrangling and preparation. With the Pandas DataFrame at the core of its API, it is very simple to use and performant on relatively small datasets. On the other hand, Apache Spark is an open-source, distributed processing system used for big data workloads. It has also gained extreme popularity as the go-to engine for prototyping and deploying Data Engineering and Machine Learning pipelines at scale. Due to the maturity and reliability that the Spark project has achieved in recent years it is widely used for production use cases in the enterprise.Like Pandas, Spark features an API with the DataFrame as a foundational data structure for analyzing data at scale. However, while it is dramatically more performant at scale than Pandas it has never been quite as easy and intuiti

After updating from CM 7.6.1 to CM 7.7.1, we are unable to connect the multiple PQS with the KNOX using the ODBC, Always getting the 500 error. But there is no issues when we are using the JDBC. The cause of the issue is due to the topology change after the CM upgrade, for services such as HIVE and AVATICA. --------------------------------------------------------------------------------------from:<value>enabled=true;maxFailoverAttempts=3;failoverSleep=1000;maxRetryAttempts=300;retrySleep=1000;</value>to:<value>enableStickySession=true;noFallback=true;enableLoadBalancing=true</value>---------------------------------------------------------------------------------------- This is in order to support the Knox load balancer feature. And we followed the below steps to revert the changes for existing cluster. Knox --> Configuration --> Knox Gateway Advanced Configuration Snippet (Safety Valve) for conf/cdp-resources.xml<name>providerConfigs:pam</name&gt

High level Problem ExplanationThe primary garbage collection challenge often arises when the heap configuration of the Namenode or Datanode is inadequate. GC pauses can trigger Namenode crash, failovers, and performance bottlenecks. In larger clusters, there is no response at the time of transitioning to an another Namenode.Detailed Technical ExplanationThe Namenode comprises three layers of management:Inode ManagementBlock ManagementSnapshot ManagementAll of these layers are stored in the Namenode in-memory heap, and any substantial modifications to them can lead to increased heap usage.Similarly, the Datanode also maintains block mapping information in its memory.Example Of Inode Reference <inode> <id>26890</id> <type>DIRECTORY</type> <name>training</name> <mtime>1660512737451</mtime> <permission>hdfs:supergroup:0755</permission> <nsquota>-1</nsquota><dsquota>-1</dsquota>&

ObjectiveThis article provides a Quickstart for using XGBoost with Spark in Cloudera Machine Learning. CML users who are looking to test a distributed version of XGBoost are welcome to use the code in their environments. The notebook is available in this Git repository. Cloudera Machine LearningCloudera AI on the Cloudera Data Platform increases data scientists' productivity by providing a single, unified platform that is all-inclusive for powering any AI use case.CML is purpose-built for agile experimentation and production ML workflows. It is designed to manage everything from data preparation to MLOps and predictive reporting. For more information on CML, please visit this page.XGBoost and SparkIn Machine Learning, Decision Tree methods are a type of Supervised Learning model. They have been used for decades in both classification and regression tasks. There are many types; generally, they are constructed by identifying ways to split data into hierarchical s

OverviewThis handbook focuses on Cloudera’s new adapter dbt-hive that enables users to use dbt on top of the Hive data warehouse system.  Links to a dbt example project that can be used to bootstrap a user’s dbt journey can be found below.Version Notes:The adapter has been tested on the following version:python: 3.9.13Cloudera Manager CM VERSION 7.6.2 RUNTIME VERSION 7.2.15Cloudera Data Warehouse on CDP v 1.4.2-b118dbt-core: 1.2.0Installing dbt and requirementsPrerequisites: Make sure python3, git, and pip are available on your system.Cloning the repo:Run the following command in your terminal to clone the example project repository:git clone https://github.com/cloudera/dbt-hive-example/Installing the packages:Run the following commands in your terminal to install the required packages/dependencies using pip/pip3:python3 -m pip install dbt-corepython3 -m pip install sqlparams==3.0.0python3 -m pip install impyla==0.18python3 -m pip install dbt-hiveCloudera Data Warehouse&