Resources

Objective Great Expectations is a Python-based open-source library for validating, documenting, and profiling your data. It helps you maintain data quality and improve communication about data between teams. Software developers have long known that automated testing is essential for managing complex codebases. Great Expectations brings the same discipline, confidence, and acceleration to data science and data engineering teams. (Source: Great Expectations) CDP Data Engineering is the only cloud-native service purpose-built for enterprise data engineering teams. Building on Apache Spark, Data Engineering is an all-inclusive data engineering toolset that enables orchestration automation with Apache Airflow, advanced pipeline monitoring, visual troubleshooting, and comprehensive management tools to streamline ETL processes across enterprise analytics teams. Data Engineering is fully integrated with the Cloudera Data Platform, enabling end-to-end visibility and security with SDX as we

ObjectiveCDP Data Engineering (CDE) is the only cloud-native service purpose-built for enterprise data engineering teams. Building on Apache Spark, Data Engineering is an all-inclusive data engineering toolset that enables orchestration automation with Apache Airflow, advanced pipeline monitoring, visual troubleshooting, and comprehensive management tools to streamline ETL processes across enterprise analytics teams.Apache Sedona™ (formerly known as "GeoSpark") is a cluster computing system for processing large-scale spatial data. Sedona extends existing cluster computing systems, such as Apache Spark and Apache Flink, with a set of out-of-the-box distributed Spatial Datasets and Spatial SQL that efficiently load, process, and analyze large-scale spatial data across machines.Sedona jobs can run in CDE with minor configuration changes. This article will present a simple workflow to help you get started with Spark Geospatial use cases in CDE.All code, dependencies and CLI commands a

What is Cloudera Data EngineeringCDP Data Engineering (CDE) is the only cloud-native service purpose-built for enterprise data engineering teams. Building on Apache Spark, Data Engineering is an all-inclusive data engineering toolset that enables orchestration automation with Apache Airflow, advanced pipeline monitoring, visual troubleshooting, and comprehensive management tools to streamline ETL processes across enterprise analytics teams.Typically CDE users leverage the CLI and API to accomplish tasks such as creating, running and orchestrating Spark and Airflow Jobs, uploading files such as Python wheel files, Jars, etc to the Virtual Cluster, and creating reusable Python environments for Spark and Airflow Jobs. CDEPY is a package that allows you to do all the above with the convenience of Python. With it you can remotely connect to a Virtual Cluster from your local machine or 3rd party tool as long as it supports Python. It is available on PyPi at this URL and can be easi

If you need to connect HiveServer2 from Third parties like Dbbeaver OR PowerBI or any Java client, you can connect using the Cloudera JDBC driver.The driver can be downloaded from Hive JDBC Connector 2.6.21 for Cloudera Enterprise. We will cover the scenario for SSL, Zookeeper,Kerberos, LDAP, and LoadBalancer to connect to HiveServer2 using the driver.    Please note that the Beeline URL is not the same as the JDBC driver. The list of properties for the JDBC driver is listed at Cloudera JDBC Driver 2.6.21 for Apache Hive.You need to use com.cloudera.hive.jdbc.HS2Driver class to connect to HiveServer2.Lets assume:  Host: c2345.node.cloudera.com  Kerberos Realm: EXAMPLE.COMPlain Connection:   jdbc:hive2://c2345.node.cloudera.com:10000/default;LogLevel=6;LogPath=/tmp    where LogLevel =6 ==> is more verbose level of loggingKerberos+SSL+binary:  jdbc:hive2://c2345.node.cloudera.com:10000/default;SS

Apache Spark is a unified analytics engine for large-scale data processing. It provides high-level APIs in Java, Scala, Python, and R, and an optimized engine that supports general execution graphs.  The below table provides you links to features released in each Spark 3 minor version: Spark Version Documentation Link Release Date Spark 3.0.0 https://spark.apache.org/docs/3.0.0/ 2020-06-16 Spark 3.0.1 https://spark.apache.org/docs/3.0.1/ 2020-11-05 Spark 3.0.2 https://spark.apache.org/docs/3.0.2/ 2021-02-19 Spark 3.0.3 https://spark.apache.org/docs/3.0.3/ 2022-06-17 Spark 3.1.1 https://spark.apache.org/docs/3.1.1/ 2021-03-02 Spark 3.1.2 https://spark.apache.org/docs/3.1.2/ 2022-06-17 Spark 3.1.3 https://spark.apache.org/docs/3.1.3/ 2022-06-17 Spark 3.2.0 https://spark.apache.org/docs/3.2.0/ 2021-10-13 Spark 3.2.1 https://spark.apache.org/docs/3.2.1/ 2022-06-17 Spark 3.2.2 https://spark.apache.org/docs/3.2.2/ 2022-07-15 Spark 3.2.3 https://spark.apache.org/

Install Install Rust from Install Rust This is interactive so needs to run in a terminalcurl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | shHello WorldRust Hello World https://doc.rust-lang.org/book/ch01-02-hello-world.html​ Reload your session and terminal to update the PATH env Runrustc main.rsmain.rs…​ Why Rust?  Original Post from Stack Overflow:  what is rust and why is it so popular Rust has been Stack Overflow’s most loved language for four years in a row, indicating that many of those who have had the opportunity to use Rust have fallen in love with it. However, the roughly 97% of survey respondents who haven’t used Rust may wonder, “What’s the deal with Rust?” Rust’s static typing does its best to get out of the programmer’s way while encouraging long-term maintainability. Rust gives you the choice of storing data on the stack or the heap and determines at compile time when memory is no longer needed and can be cleaned up. The biggest benefit Rust c

Introduction The new method of capturing Cloudera Manager performance profile utilizes the Java Flight Recorder (JFR) feature of JVM. (Running Java Flight Recorder)  JFR works on OpenJDK 11 and higher,  also OpenJDK 8u262 and higher.  For older JDK8, use the older method for capturing performance profiles. (Steps to collect Cloudera Manager profiling data to troubleshoot performance issues - you need subscription to access this article) The performance capture files can be analyzed with the JMC tool - JDK Mission Control (JMC)  Pre-requisites Ensure recent JDK # /usr/java/default/bin/java -versionopenjdk version "1.8.0_232" OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_232-b09) OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.232-b09, mixed mode) JDK 8u232 is not recent enough to support JFR.  Will not work.  Must be OpenJDK 8u262 or higher. —--- # /usr/java/default/bin/java -versionopenjdk version "11.0.3" 2019-04-16 LTS OpenJDK Runt

Hey Cloudera Community!We're thrilled to announce our latest addition to the Cloudera AMP catalog - Text Summarization and more with Amazon Bedrock! If you've been keen on tapping into the power of foundation models from leading AI companies like Amazon, this AMP is crafted just for you.What are AMPs?Applied ML Prototypes (AMPs) are pre-packaged ML projects designed for easy deployment within Cloudera AI. AMPs enable data scientists to go from an idea to a fully working ML use case in a fraction of the time.What's Inside:Guided Setup: Navigate effortlessly through the setup process and start using the Amazon Bedrock API in no time.Highlighted Models: Dive deep with two powerful models:amazon.titan-tg1-largeanthropic.claude-v2 (Supports a whopping 100K tokens in the prompt!)Flexibility in Action: Customize the instruction prompts as needed to instruct the models for diverse text generation tasks.API Guidance: Benefit from clear directions on prompt formats and request API schema for has

Synopsis.ORC is a columnar storage format for Hive.This document is to explain how creation of ORC data files can improve read/scan performance when querying the data. TEZ execution engine provides different ways to optimize the query, but it will do the best with correctly created ORC files.ORC Creation Strategy.Example:CREATE [EXTERNAL] TABLE OrcExampleTable(clientid int, name string, address string, age int)stored as orcTBLPROPERTIES ("orc.compress"="ZLIB","orc.compress.size"="262144","orc.create.index"="true","orc.stripe.size"="268435456","orc.row.index.stride"="3000","orc.bloom.filter.columns"="clientid,age,name");Ingesting data into Hive tables heavily depends on usage patterns. In order to make queries running efficiently, ORC files should be created to support those patterns. -Identify most important/frequent queries that will be running against your data set (based on filter or JOIN conditions)-Configure optimal data file size-Configure stripe and stride size-Distribute and so

Pandas is a fast, powerful and flexible Python data analysis and manipulation tool. It has gained incredible popularity in the Data Science community as one of the most important  packages for data wrangling and preparation. With the Pandas DataFrame at the core of its API, it is very simple to use and performant on relatively small datasets. On the other hand, Apache Spark is an open-source, distributed processing system used for big data workloads. It has also gained extreme popularity as the go-to engine for prototyping and deploying Data Engineering and Machine Learning pipelines at scale. Due to the maturity and reliability that the Spark project has achieved in recent years it is widely used for production use cases in the enterprise.Like Pandas, Spark features an API with the DataFrame as a foundational data structure for analyzing data at scale. However, while it is dramatically more performant at scale than Pandas it has never been quite as easy and intuiti

After updating from CM 7.6.1 to CM 7.7.1, we are unable to connect the multiple PQS with the KNOX using the ODBC, Always getting the 500 error. But there is no issues when we are using the JDBC. The cause of the issue is due to the topology change after the CM upgrade, for services such as HIVE and AVATICA. --------------------------------------------------------------------------------------from:<value>enabled=true;maxFailoverAttempts=3;failoverSleep=1000;maxRetryAttempts=300;retrySleep=1000;</value>to:<value>enableStickySession=true;noFallback=true;enableLoadBalancing=true</value>---------------------------------------------------------------------------------------- This is in order to support the Knox load balancer feature. And we followed the below steps to revert the changes for existing cluster. Knox --> Configuration --> Knox Gateway Advanced Configuration Snippet (Safety Valve) for conf/cdp-resources.xml<name>providerConfigs:pam</name&gt

High level Problem ExplanationThe primary garbage collection challenge often arises when the heap configuration of the Namenode or Datanode is inadequate. GC pauses can trigger Namenode crash, failovers, and performance bottlenecks. In larger clusters, there is no response at the time of transitioning to an another Namenode.Detailed Technical ExplanationThe Namenode comprises three layers of management:Inode ManagementBlock ManagementSnapshot ManagementAll of these layers are stored in the Namenode in-memory heap, and any substantial modifications to them can lead to increased heap usage.Similarly, the Datanode also maintains block mapping information in its memory.Example Of Inode Reference <inode> <id>26890</id> <type>DIRECTORY</type> <name>training</name> <mtime>1660512737451</mtime> <permission>hdfs:supergroup:0755</permission> <nsquota>-1</nsquota><dsquota>-1</dsquota>&

ObjectiveThis article provides a Quickstart for using XGBoost with Spark in Cloudera Machine Learning. CML users who are looking to test a distributed version of XGBoost are welcome to use the code in their environments. The notebook is available in this Git repository. Cloudera Machine LearningCloudera AI on the Cloudera Data Platform increases data scientists' productivity by providing a single, unified platform that is all-inclusive for powering any AI use case.CML is purpose-built for agile experimentation and production ML workflows. It is designed to manage everything from data preparation to MLOps and predictive reporting. For more information on CML, please visit this page.XGBoost and SparkIn Machine Learning, Decision Tree methods are a type of Supervised Learning model. They have been used for decades in both classification and regression tasks. There are many types; generally, they are constructed by identifying ways to split data into hierarchical s

OverviewThis handbook focuses on Cloudera’s new adapter dbt-hive that enables users to use dbt on top of the Hive data warehouse system.  Links to a dbt example project that can be used to bootstrap a user’s dbt journey can be found below.Version Notes:The adapter has been tested on the following version:python: 3.9.13Cloudera Manager CM VERSION 7.6.2 RUNTIME VERSION 7.2.15Cloudera Data Warehouse on CDP v 1.4.2-b118dbt-core: 1.2.0Installing dbt and requirementsPrerequisites: Make sure python3, git, and pip are available on your system.Cloning the repo:Run the following command in your terminal to clone the example project repository:git clone https://github.com/cloudera/dbt-hive-example/Installing the packages:Run the following commands in your terminal to install the required packages/dependencies using pip/pip3:python3 -m pip install dbt-corepython3 -m pip install sqlparams==3.0.0python3 -m pip install impyla==0.18python3 -m pip install dbt-hiveCloudera Data Warehouse&

Issue During the installation of the latest version of mlflow for the custom runtime engine in CDSW 1.10.x, it's still using cmladdon’s version rather than the version installed via pip. For example, in one case, the error recorded was the following: ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflict mlflow-cml-plugin 0.0.1 requires protobuf==3.19,4, but you have protobuf 4.24.0 which is incompatible. 
onnx 1.12.0 requires protobuf<=3.20.1,>=3.12.2, but you have protobuf 4.24.0 which is incompatible.ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflict mlflow-cml-plugin 0.0.1 requires protobuf==3.19,4, but you have protobuf 4.24.0 which is incompatible. 
onnx 1.12.0 requires protobuf<=3.20.1,>=3.12.2, but you have protobuf 4.24.0 which

Hi Everyone, I ran into a situation where I have to the following XML Input:     <workspace id="1234"><documents totalCount="2"><document id="5555" title="document1" revision="1"><file name="file1.pdf" type="pdf"/><attributes> <attribute name="custAtt1" value="v1"/> <attribute name="custAtt2" value="v2"/> <attribute name="custAtt3" value=""/></attributes></document><document id="6666" title="document2" revision="0"> <file name="file2.xlsx" type="xlsx"/> <attributes> <attribute name="custAtt1" value="v1"/> <attribute name="custAtt2" value="v2"/> <attribute name="custAtt3" value="v3"/> </attributes></document></documents></workspace>     Here, each /workspace/documents/document record needed to be split and transformed into JSON, where: Each document object needs to have the workspace ID it belongs to. Each document attribute value will be as

Introduction: This article is designed to get the reader to call a deployed CML model from Apache NiFi or Cloudera Data Flow as quickly as possible.  It will require using one of Cloudera's Applied Machine Learning Prototypes (AMPs) on Customer Churn and the Apache NiFi Flow provided in this article.  (Git Repo Here) Prerequisites: Apache NiFi instance is running. CML's AMP Customer Churn App is running with a Churn Model deployed.  Download the user data file and Apache Nifi flow from this repo.  The user data file needs to be added to the file location where NiFi's GetFile Processor has permissions.  Launch CML Churn AMP In the AMP catalog, select Churn Modeling with skit-learn, and configure a new project.  This will take 5-7 minutes to complete.  You will need to deploy a model inside of CML, which isn't in the scope of this article.  Once you finish, you will be able to see the following screen, which will provide the URL location an

In this article, we will learn how to run scheduled ETL workloads on CDE ( Cloudera Data Engineering )  and CDW ( Cloudera Data Warehouse) using Airflow.   CDE 1.4 release provides the capability to schedule Airflow jobs (shipped along with CDE). If we need to learn more about Airflow, refer to the Airflow website.  This release adds two Airflow operators: CDEJobRunOperator and CDWOperator. The CDEJobRunOperator behind the scene calls the jobs API of CDE to run Spark jobs. More details on jobs API can be found here.   The CDWOperator behind the scene makes use of JDBC connection to interact with CDW Hive Virtual warehouse or Impala Virtual warehouse.   In this article to demonstrate ETL workload, we will be reading CSV from AWS S3 bucket using Spark, perform a transformation (via CDEJobRunOperator) and load the processed data to CDW warehouse using Airflow CDWOperator. Pre-Requisite  CDP environment is created and CDW, CDE experience are provisio

This is a simple command that can be used to run via a browser to change the log level without needing a restart.For Hbase Master:http://<SERVER-HOSTNAME>:16010/logLevel?log=org.apache.hadoop.hbase& level=<DESIRED-LOG-LEVEL>For HBase Region Server:http://<SERVER-HOSTNAME>:16030/logLevel?log=org.apache.hadoop.hbase& level=<DESIRED-LOG-LEVEL> **NOTE:Please make sure there is no space between '&' and 'level'. There seems to be a formatting issue in HCC because of which a space was introduced in the syntax.<SERVER-HOSTNAME> : Represents the hostname of the server where the daemon is running.<DESIRED-LOG-LEVEL>: Represent the log level that is desired by the user to have. Ex. INFO, DEBUG,..

Spark3 HBase Integration This blog post will guide you through the process of integrating Spark 3 with HBase, providing you with valuable insights and step-by-step instructions.Note: This blog post will cover both Spark3 and HBase services only when they exists in the same cluster. If HBase is in remote  cluster, you can go through the hbase-configure-spark-connector-remote-cluster tutorial.PrerequisitesTo use HBase-Spark3 connector, we need to ensure that CDP version is 7.1.7 SP1 Spark3 parcel or above.Ensure that every Spark node has the HBase Master, Region Server, or Gateway role assigned to it. If no HBase role is assigned to a Spark node, add the HBase Gateway role to it, which ensures that the HBase configuration files are available on the Spark node.HBase Integration with Spark3HBase integration with Spark can be achieved using the HBase-Spark Connector, which provides a seamless way to interact with HBase from within Spark applicati

Description Repo Name: cml_vscode_with_python310Repo Info: VsCode using Python 3.10, instead of the existing VsCode example using Python 3.7. Repo URL: https://github.com/ryancicak/cml_vscode_with_python310 

I recently received a question "We'd like to have Python and R installed in the docker image for a package.  Is it possible to have such customized docker with python >=3.7, R>=4.0 and CUDA==11.7?" I'd like to go through how I accomplished this using the pbj-workbench-r4.1-standard.Dockerfile.  This runtime has the essentials (Python and R) except for CUDA (NVIDIA GPU).  Therefore, we'll use pbj-workbench-r4.1-standard.Dockerfile as a base image.  It's important to note that CML requires a runtime kernel that needs to be either R or Python, it can't be both.  I'll go through three different iterations of this image: Iteration 1 - Since this image already has R and Python installed, we'll change the runtime NOT to use Rryancicak/pbj_r-and-python:v9 -> Python 3.10 (GitHub Tag: python/3.10/standard_dockerhub_v9)https://github.com/ryancicak/pbj_r-and-python/blob/python/3.10/standard_dockerhub_v9/Dockerfile Iteration 2 - G

Spark Streaming Graceful shutdownSpark Streaming jobs are long-running jobs and their tasks need to be executed 7*24 hours, but there are cases like upgrading the code or updating the Spark configuration, where the program needs to be actively stopped. However, for distributed programs, there is no way to kill processes one by one. It is very important to shut down all configurations gracefully.There are two ways to shut down streaming applications:Non-graceful shutdownGraceful shutdown1. Non-graceful shutdownA non-graceful shutdown is a violent shutdown process. Stopping Spark streaming violently may cause problems. For example, if your data source is Kafka, and a batch of data has been loaded into Spark streaming for processing. If it is forced to stop, consumption will be repeated or some data will be lost at the next startup. Disadvantages:Spark Streaming works based on the micro-batch mechanism, RDD is generated according to the interval time. If a violent shut

OBJECTIVE:Provide a quick-start guide for using the Jolt language within a NiFi JoltTransform (JoltTransformJSON or JoltTransformRecord).OVERVIEW:The NiFi JoltTransform uses the powerful Jolt language to parse JSON. Combined with the NiFi Schema Registry, this gives NiFi the ability to traverse, recurse, transform, and modify nearly any data format that can be described in AVRO or, using JSON as an intermediary step.Although the language itself is open-source, and some documentation is available in the JavaDoc, this article can provide a starting point for understanding basic Jolt operations.PREREQUISITES:HDF 3.0 or later (NiFi 1.2.0.3 or later)BASICS OF JOLT:Simplified OverviewThe JoltTransform applies a set of transformations described in a JSON specification to an input JSON document and generates a new output JSON document.Jolt SpecificationOverviewA Jolt Specification is a JSON structure that contains two root elements:operation (string): shift, sort, cardinality, modify-default-b

Starting from NiFi 1.14, you can specify the TLS ciphers to be used by NiFi web service by using below property: nifi.web.https.ciphersuites.include You can also specify the TLS Ciphers to be excluded by using below property: nifi.web.https.ciphersuites.exclude This enhancement is part of Apache Jira NIFI-3580.    For Cloudera CFM, these properties are exposed (and usable) from CFM 2.1.3 and above.   Note: The values for above properties should take the form of a comma-separated list of common cipher names as specified here. The regular expressions (for example ^.*GCM_SHA256$) may also be specified.