Community Articles

Background: This is a new article from the Partner Solutions Engineering Team describing how to connect to CDW Hive using ODBC on your Mac. In this series, we will explore client-side activities and Partner integrations that may sometimes be atypical. Our goal is to offer convenient approaches that reduce time and research on your part. Step 0: Ensure that homebrew and keytool are installed on your Mac and in your path.% export PATH=$PATH:/opt/homebrew/bin% which keytool/opt/homebrew/opt/openjdk/bin/keytool Step 1: Download the CDW certificates for the Hive virtual cluster you want to connect to. Assuming you downloaded the certificate chain into a file called truststore.jks (can be any name you want), create a PEM file:% keytool -list -rfc -keystore truststore.jks | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > cdw-certs.pemYou should see multiple certificates in this file. Please also note that the /etc/hosts file on your ODBC client should be able to resolve the virtu

Purpose: Run SELECT to ingest data from Oracle 19c, and save the data into Azure ADLS Gen2 object storage, in Parquet format. Steps Step 1 Prepare the environment Make sure the Oracle 19c environment works well.  Prepare an Oracle table:    CREATE TABLE demo_sample ( column1 NUMBER, column2 NUMBER, column3 NUMBER, column4 VARCHAR2(10), column5 VARCHAR2(10), column6 VARCHAR2(10), column7 VARCHAR2(10), column8 VARCHAR2(10), column9 VARCHAR2(10), column10 VARCHAR2(10), column11 VARCHAR2(10), column12 VARCHAR2(10), CONSTRAINT pk_demo_sample PRIMARY KEY (column1, column2, column3, column4, column5, column6, column7, column8, column9));   Prepare 20

IntroductionIn this previous article, we talked about the benefits of deploying COD on Cloud Storage, covering the challenges involved with maintaining the consistency of HBase data and the different attempts to solve that problem, with a detailed technical description of the Store File Tracking (SFT) feature we've now been using for COD. Now we can successfully move our dataset to the cloud storage file system of preference, decoupling compute from storage and benefiting from optimal resource usage and independent scale. Moving the data away from RegionServers, however,  comes with a (performance) cost: the additional latency overhead of having to access the cloud storage. Our initial benchmarks showed that COD over cloud storage performed, on average, five times slower than COD clusters using HDFS on block storage with HDD (for a specific vendor).  HBase already ships built-in cache capabilities, including support for file-based caching. Also all major cloud provi

A Hive_on_Tez job goes through the following stages during its execution Query Submission When a user submits a Hive query, either through the Hive command line interface (CLI), HiveServer2, or through an application such as Hue, the query is sent to the Hive service for processing Query Parsing and Compilation The Hive service parses the submitted query to understand its structure and requirements. It then compiles the query into an execution plan specifying the steps needed to execute it. This execution plan includes details such as the sequence of Tez tasks required to perform the computation. Tez Session Initialization Once the optimized execution plan is ready, Hive initializes a Tez session. The Tez session provides a runtime environment for executing the query using Apache Tez, a framework optimized for processing large-scale data. This initialization process involves setting up necessary configurations, loading required libraries, and establishing communication channels with th

CDE is the Cloudera Data Engineering Service, a containerized managed service for the Cloudera Data Platform designed for Large Scale Batch and Streaming Pipelines with Spark, Airflow, and Iceberg. It allows you to submit batch jobs to auto-scaling virtual clusters. As a Cloud-Native service, CDE enables you to spend more time on your applications, and less time on infrastructure. A CDE Session is an interactive short-lived development environment for running Python, Scala, and Spark commands to help you iterate upon and build your Spark workloads. You can use CDE Sessions in CDE Virtual Clusters of type "All Purpose - Tier 2". A resource in CDE is a named collection of files used by a job or a session. Resources can include application code, configuration files, custom Docker images, and Python virtual environment specifications As of version 1.20, CDE introduces the ability to leverage CDE Resources when running CDE Interactive Sessions. This means that you can leverage fil

Introduction Cloudera Operational Database (COD) serves as a foundational service within the Cloudera Data Platform (CDP), enabling users to effortlessly create operational databases that dynamically scale to meet workload demands. When deploying high-performance applications at scale, a robust operational database plays a crucial role. COD addresses this need by offering a highly scalable and high-performance operational database engineered to support data-intensive applications. Leveraging the robust foundations of Apache HBase and Apache Phoenix, COD is integrated into the Cloudera Data Platform (CDP) in the public cloud. It is designed for versatility, accommodating hybrid as well as multi-cloud deployments, ensuring adaptability across various cloud environments including Amazon AWS, Microsoft Azure, and Google GCP. For AWS deployments, COD provides two primary storage options: S3 with Ephemeral Cache, known for its high performance albeit with slightly higher costs, and S3 withou

We will be using the Ozone ldb CLI tool to view the RocksDb table, which stores all the information in various tables. Here I will be showing how to check the table data that it has created for the filesystem in Ozone. [root@ccycloud-1 data]# pwd/var/lib/hadoop-ozone/scm/data[root@ccycloud-1 data]# pwd/var/lib/hadoop-ozone/om/data We need to traverse to the Ozone metadata directory through Cloudera Manager UI > configuration and search for "metadata". [root@ccycloud-1 data]# ozone debug ldb --db=om.db lsdefaultfileTableprincipalToAccessIdsTabledeletedTableuserTables3SecretTabletransactionInfoTableopenKeyTablesnapshotInfoTabledirectoryTableprefixTablemultipartInfoTablevolumeTabletenantStateTabledeletedDirectoryTabletenantAccessIdTableopenFileTablesnapshotRenamedTabledTokenTablemetaTablekeyTablebucketTable  Under the directory /var/lib/hadoop-ozone/scm/data we need need to ls to list out the tables.[root@ccycloud-1 data]# ozone debug ldb --db=scm.db lsde

Introduction Cloudera Operational Database (COD) is a service that runs on the Cloudera Data Platform (CDP). COD enables you to create a new operational database that automatically scales based on your workload. To deploy high-performance applications at scale, a rugged operational database is essential. COD is a high-performance and highly scalable operational database designed for powering, at any scale, the biggest data applications on the planet. Powered by Apache HBase and Apache Phoenix, COD ships out of the box with Cloudera Data Platform (CDP) in the public cloud. It’s also ready for hybrid and multi-cloud deployments to meet your business where it is today, whether AWS, Microsoft Azure, or GCP. Support for cloud storage is an important capability of COD that, in addition to the pre-existing support for HDFS on local storage, offers customers a choice of price-performance characteristics. Please refer to the blog for more information on the performance differences between COD o

IntroductionApache HBase provides different cache capabilities. In Cloudera Operational Database (COD) deployments using cloud storage, we leverage the file-based BucketCache implementation to deploy it over ephemeral SSD volumes to cache the whole user dataset to avoid client reads from having to reach the slower cloud storage layer. However, the file-based BucketCache implementation was originally volatile, meaning the cache was not retained upon region server restarts. This requires a warm-up period every time a region server is restarted, to maintain optimal read performance. Depending on the data size and the configured cache size, this warm-up can take anywhere from a few minutes to a few hours. To eliminate this, the Cloudera team and the HBase community implemented the BucketCache persistence feature (HBASE-27264/HBASE-27486), where the region servers periodically persist the blocks cached in the bucket cache. This persisted information is then used to recover the cache in the

Simple API to find region name for given row key. It will also give you hint about the region - Where key will be placed.To compile class:$JAVA_HOME/bin/javac -cp `hbase classpath`: Regionfinder.javaTo run tool:$JAVA_HOME/bin/java -cp `hbase classpath`: Regionfinder <tablename> <rowkey>Regionfinder.javaimport java.io.IOException;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.hbase.HBaseConfiguration;import org.apache.hadoop.hbase.HRegionLocation;import org.apache.hadoop.hbase.TableName;import org.apache.hadoop.hbase.client.Connection;import org.apache.hadoop.hbase.client.ConnectionFactory;import org.apache.hadoop.hbase.client.Get;import org.apache.hadoop.hbase.client.RegionLocator;import org.apache.hadoop.hbase.client.Result;import org.apache.hadoop.hbase.client.Table;import org.apache.hadoop.hbase.util.Bytes;public class Regionfinder { private static void usage(String[] args) { if (args.length < 2) { Sys

Earlier this year Cloudera AI added a new way to accelerate GenAI projects by tapping into Hugging Face Spaces and deploying these projects right inside of CML with just a few clicks. With over 6,500 spaces as of this writing, Hugging Face community is still growing rapidly and provides a convenient platform for practitioners and organizations to share their work in areas from classical machine learning to the latest GenAI research. In this article you will learn how to enable and use this feature to accelerate your own ML projects.The default Hugging Face Spaces AMP catalog is enabled for all CML Public Cloud workspaces starting from version 2.0.43-b208. To enable users to launch external Hugging Face AMPs, additional steps are necessary (see end of this article).Steps to Deploy Hugging Face Space AMPLet's dive right in and see how simple it is to deploy a Hugging Face AMP:Click on AMPs in the left sidebar of your ML Workspace. If you don’t see this, then AMPs are not enabled by your

Overview Once Data Engineers complete the development and testing of their workloads, moving to deployment can be a challenging task. This process often involves a lengthy checklist, including the modification of multiple configurations that differ across environments. For instance, the configuration spark.yarn.access.hadoopFileSystems must now point to the Data Lake or Data Store appropriate for the production environment. Data Engineers may also opt to use a "staging" environment to test job performance, necessitating changes to these configurations twice: initially from development to staging, and subsequently for production deployment. A recommended practice in such cases is to use templating the configurations within the workload. This approach allows for easier management of deployments across different environments. In this article, we demonstrate a pattern of templating Data Engineering workloads in Cloudera Data Engineering in Airflow. This best practice allows workload deploy

Introduction Apache Hadoop's efficiency and reliability significantly depend on the Java Virtual Machine (JVM)'s performance. To assist in monitoring and analyzing JVM pauses, especially those induced by Garbage Collection (GC), Hadoop integrates a utility named JvmPauseMonitor, a Hadoop common class introduced by https://issues.apache.org/jira/browse/HADOOP-9618 and widely used in Hadoop projects like HDFS, Hbase, Yarn, Oozie, MapReduce, etc. Overview of JvmPauseMonitor Link to reference code here Key Features and Implementation     private String formatMessage(long extraSleepTime, Map<String, GcTimes> gcTimesAfterSleep, Map<String, GcTimes> gcTimesBeforeSleep) { Set<String> gcBeanNames = Sets.intersection( gcTimesAfterSleep.keySet(), gcTimesBeforeSleep.keySet()); List<String> gcDiffs = Lists.newArrayList(); for (String name : gcBeanNames) { GcTimes diff = gcTimesAfterSleep.get(name).subtract( gcTimes

This article explains how to configure Spark Connections in Cloudera Machine Learning. CML enables easy Spark data connections to the data stored in the Data Lake by abstracting the SparkSession connection details. CML users don't need to set complicated endpoint and configuration parameters or load HWC or Iceberg libraries. They can use the cml.data library to get preconfigured connections. CML Data Connection Snippets  The spark SparkSession object has all the necessary options set to make connections work. Users have two options to set custom or job-specific configurations like executor CPU and memory parameters. 1. Specify the configuration inlineWith SparkContext's setSystemProperty method, you can set Spark properties that will be picked up while building the SparkSession object. You can set any Spark property like below before calling cmldata's get_spark_session method.   Python script import cml.data_v1 as cmldatafrom pyspa

While running Hive query from Hue, the query fails with the following error:WARN [HiveServer2-Handler-Pool: Thread-41]: thrift.ThriftCLIService (ThriftCLIService.java:FetchResults(596)) - Error fetching results: org.apache.hive.service.cli.HiveSQLException: Couldn't find log associated with operation handle:OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=1cd26ef5-00f8-4003-be5c-65e2a542ee18] at org.apache.hive.service.cli.operation.OperationManager.getOperationLogRowSet (OperationManager.java:257) at org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:656) at sun.reflect.GeneratedMethodAccessor104.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:79) at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionPro

JDBC/ODBC Drivers for Hive can be downloaded from Cloudera website.The first thing we need to collect is the Hive Endpoint from Cloudera Management console. This can be found at the bottom of in the specific DataHub window. It will be in the following format: jdbc:hive2://datahub1-master0.geo-1035.lskx-pvue.a4.cloudera.site/;ssl=true;transportMode=http;httpPath=datahub1/cdp-proxy-api/hive Majorly we need to furnish the following information in the appropriate fields:Host(s) : datahub1-master0.geo-1035.lskx-pvue.a4.cloudera.sitePort : 443Authentication Mechanism : Username/PasswordThrift Transport : HTTPGo to HTTP Options :  HTTP Path : datahub1/cdp-proxy-api/hive  (you will get this info from the Hive Endpoint)Go to SSL Options:6.1. Check Enable SSL6.2. Check Allow Self-signed Server Certificate check box6.3. Trusted Certificates: Select the path to the PEM file containing the root ca cert of the Knox gateway.Note: You can download

Log Forwarding/Ingestion PatternsLog forwarding & ingestion is a key starting point for many logging initiatives such as log analytics, cyber security, anomaly & bot detection, etc etc. This article will focus few (not comprehensive) patterns for log forwarding/ingestion using NiFi. Commonly rsyslog is used to capture and ship log messages.“Rsyslog is an open-source software utility used on UNIX and Unix-like computer systems for forwarding log messages in an IPnetwork. It implements the basic syslog protocol, extends it with content-based filtering, rich filtering capabilities, flexible configuration options and adds features such as using TCP for transport.”More on how to configure rsyslog: hereNiFi is able to ingest messages from rsyslog over TCP or UDP via ListenSysLog processor. This allows for little to no coding. Patterns Pattern A A minimalist design. Rsyslog is configured to simply forward log messages to a NiFi cluster. Rsyslog /etc/rsyslog.conf file needs to be conf

Cloudera Data EngineeringCDE is the Cloudera Data Engineering Service, a containerized managed service for Cloudera Data Platform designed for Large Scale Batch Pipelines with Spark, Airflow and Iceberg. It allows you to submit batch jobs to auto-scaling virtual clusters. As a Cloud-Native service, CDE enables you to spend more time on your applications, and less time on infrastructure.CDE allows you to create, manage, and schedule Apache Spark jobs without the overhead of creating and maintaining Spark clusters. With CDE, you define virtual clusters with a range of CPU and memory resources, and the cluster scales up and down as needed to run your Spark workloads, helping to control your cloud costs.Cloudera Data Engineering provides a command line interface (CLI) client. You can use the CLI to create and update jobs, view job details, manage job resources, run jobs, and more.Apache IcebergApache Iceberg is a cloud-native, high-performance open table format for organizing petabyte-scal

Installing Spark3 and Livy3 on Cloudera Manager with CDS3 ParcelApache Spark Logo 1. IntroductionThis guide outlines the steps for installing Apache Spark 3.x on your Cloudera cluster by leveraging Cloudera Manager (CM) and CDS3 parcels. Learn how to efficiently download, distribute, and activate the CDS3 parcel for a seamless Spark3 deployment, saving you time and effort compared to traditional methods. Additionally, the guide provides resources for troubleshooting any potential issues encountered during the installation.Note: This article, mainly focuses on Cloudera Manager Server does have Internet access.2. CDP-compatible CDS3 Parcels DetailsCDS 3 Powered by Apache Spark is an add-on service for CDP Private Cloud Base, distributed as a parcel. The Cloudera Service Descriptor(CDS) file is available in Cloudera Manager for CDP 7.1.X.The CDS version label is constructed in v.v.v.w.w.xxxx.y-z...z format and carries the following information:v.v.v - Apache Spark upst

In this article, I will demonstrate how to access the Ozone file system using Java API. I  explain the steps to validate the Ozone cluster for volume, bucket, and key creation. Let's begin and create a Java project with Maven build:  Create a Java project with the name AccessOFS using IntelliJ from File->New->Project.The project structure will look like the below image: Modify your pom.xml and add the dependencies for ozone-client. The pom.xml will be as follows.<?xml version="1.0" encoding="UTF-8"?><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>org.demo</groupId> <artifactId>AccessOFS</artifactId> <version>1.0-SNAPSHOT</version> <build> <plugins> <

In this article I am going to review the required steps and processes to setup some NiFi SSL Context Services with modern versions of NiFi (1.20, 1.21, 2.0).    In the past, nifi installations did not come installed with SSL enabled.   If you were a NiFi admin and had to setup ssl, you know it was not an easy task, and could often times prevented the cluster from even being secured at all.   Thanks to the wonderful team of innovators working on the Apache NiFi Project, I am happy to show the Easy Button works to install a fully secured Nifi.  I will also show how to setup SSL Context Services for internal and external connection to https enabled endpoints. First, let's do a new install of Apache NiFi 1.21.   This article is not about how to do that, so fast forward to a running NiFi,  lets take a look at some important details that you will find in the nifi-app.log: Your login details:2023-06-01 10:02:55,493 INFO [main] o.a.n.

Logging Iceberg Metrics with MLFlow Tracking in CMLCloudera AI on Cloudera Data Platform (CDP) accelerates time-to-value by enabling data scientists to collaborate in a single unified platform that is all inclusive for powering any AI use case. Purpose-built for agile experimentation and production ML workflows, Cloudera Machine Learning manages everything from data preparation to MLOps, to predictive reporting.CML is compatible with the MLflow Tracking API and makes use of the MLflow client library as the default method to log experiments. Existing projects with existing experiments are still available and usable. CML’s experiment tracking features allow you to use the MLflow client library for logging parameters, code versions, metrics, and output files when running your machine learning code.Apache Iceberg is a table format for huge analytics datasets in the cloud that defines how metadata is stored and data files are organized. Iceberg is also a library that compute engines can use

CML Model Deployment with MLFlow and APIv2Cloudera AI on Cloudera Data Platform (CDP) accelerates time-to-value by enabling data scientists to collaborate in a single unified platform that is all inclusive for powering any AI use case. Purpose-built for agile experimentation and production ML workflows, Cloudera Machine Learning manages everything from data preparation to MLOps, to predictive reporting.CML exposes a REST API that you can use to perform operations related to projects, jobs, and runs. You can use API commands to integrate CML with third-party workflow tools or to control CML from the command line.In this example we will showcase how to use APIv2 to programmatically register an XGBoost experiment via MLFlow Tracking, Registry, and deploy it as a model endpoint in CML.MLOps in CMLCML has extensive MLOps features and a set of model and lifecycle management capabilities to enable the repeatable, transparent, and governed approaches necessary for scaling model deployments and

PurposeDetect updates to S3 files and insert the updated files into Aurora PostgreSQL with NiFiData flowThe finished dataflowProcess1.) Download the dataflow file(JSON file Import_S3_To_Aurora_PostgreSQL.json)2.) Create a new processor group.When creating this processor group, choose the following JSON file to upload.Step 1: Choose processor group Step 2 : Step 3 : Finish upload. 3.) Install JDBC Driverwget https://jdbc.postgresql.org/download/postgresql-42.7.3.jarmkdir /tmp/nifimv postgresql-42.7.3.jar /tmp/nifi/4.) Set parameters in NiFiSet ListS3 parameters  S3 Access Key set    The values input was protected as access key/values are sensitive values.Only "Sensitive value set" be shown. 4.2) Start the controllver service AWSCredentialsProviderControllerService (for saving AWS sensitive values)  4.3) Start the CSVReader Controller service   4.4) Start the JDBC Connection pool（DBCPConnectionPool-postgreSQL）service 5. ) Set bu

When we develop applications for the Cloudera Data Platform, it is quite often necessary to use third-party libraries like NumPy, SciPy, Pandas, etc, or even different versions of existing components, such as Python.  On the other hand, installing and maintaining Python environments can be a complex and time-consuming task for a system administrator and extra caution needs to be taken when we talk about installing other versions of Python and its modules at the operating system level due to the requirements of Cloudera Manager and Cloudera Runtime. You could also choose to install Python virtual environments, but that would still require effort to keep all cluster nodes up to date. Running python in virtual environments in YARN mode requires extra development effort and a significant increase in the total application size due to dependencies. Another excellent option is to distribute Anaconda as a parcel, but be aware that generating custom parcels requires the Anaconda Enterpri