Resources

Each rule in the auth-to-local rules as the following formats:RULE:[n:string](regexp)s/pattern/replacement/RULE:[n:string](regexp)s/pattern/replacement/gRULE:[n:string](regexp)s/pattern/replacement//LRULE:[n:string](regexp)s/pattern/replacement/g/L [n:string]Indicates a matching rule where n declares the number of expected components in the principal. Components are separated by a /, where a user account has one component (ambari-qa) and a service account has two components (nn/fqdn). The string value declares how to reformat the value to be used in the rest of the expression. The placeholders are as follows:$0 - realm$1 - 1st component$2 - 2nd componentTypically we ignore the 2nd component since it is the service’s hostname and thus the format is generally set to $1@$0 (but can be any pattern) as in:[1:$1@$0]Matches on ambari-qa@EXAMPLE.COMTranslates to ambari-qa@EXAMPLE.COM[2:$1@$0]Matches on nn/c6501.ambari.apache.org@EXAMPLE.COMTranslates to nn@EXAMPLE.COM (regexp)Indicates a

Prerequisite :1. We should have the fsimage to continue with this analysis.2.Access to git What is small files ?A small file is one which is significantly smaller than the default Apache Hadoop HDFS default block size (128MB by default in CDH/CDP). One should note that it is expected and inevitable to have some small files on HDFS. These are files like library jars, XML configuration files, temporary staging files, and so on. But when small files become a significant part of datasets, the problems arise.Problems with small files and HDFSSmall files are a common challenge in the Apache Hadoop world and when not handled with care, they can lead to a number of complications. The Apache Hadoop Distributed File System (HDFS) was developed to store and process large data sets over the range of terabytes and petabytes. However, HDFS stores small files inefficiently, leading to inefficient Namenode memory utilisation and RPC calls, block scanning throughput degradation, and reduced applic

IntroductionApache Iceberg HistoryApache Spark and Iceberg Supportability Matrix TableReferences Introduction The Spark and Iceberg Supportability Matrix provides comprehensive information regarding the compatibility and supportability of Spark and Iceberg versions with various operating systems, frameworks, and dependencies. Apache Iceberg History The development of Iceberg was started in 2017 by Netflix. The project was open-sourced and donated to the Apache Software Foundation in November 2018. In May 2020, the Iceberg project graduated to become a top-level Apache project. Apache Iceberg 0.7.0 was released on Oct 26, 2019 (Incubating) Apache Iceberg 0.8.0 was released on May 7, 2020 (Incubating). Apache Iceberg 0.9.0 was released on Jul 14, 2020. Apache Iceberg 0.9.1 was released on Aug 11, 2020. Apache Iceberg 0.10.0 was released on Nov 12, 2020. Apache Iceberg 0.11.0 was released on Jan 27, 2021. Apache Iceberg 0.11.1 was released on Apr 3, 2021. Apache Iceberg 0

  In this article, I will document how to use CFM 1.0.1.0 to interact with Apache Impala. This article still applies if using HDF / Apache NiFi   The latest official JDBC driver that will work when using NiFi is the  JDBC driver 2.6.4 or below. At the time of this writing, any driver above that causes class conflicts with the NiFi JVM and the driver's own use of log4j.   Pre-requisite   Downloading and extracting the JDBC driver:   The JDBC drivers can be found at Impala JDBC Connector 2.6.15 for Cloudera Enterprise. Select 2.6.4 * or if in the future a version higher than 2.6.15 is available, use that. Download and extract 2.6.4 and make note of where it extracts to. Ensure that the user that runs the NiFi JVM ( nifi ) has permission to that path. The jar file that you will use is called ImpalaJDBC41.jar.   Create Impala table and load dataset sample to HDFS:   Use this data set tips.csv and add it to your HDFS. hdfs dfs

PurposeBuilding Custom Operators helps Data engineers abstract and reuse code across workloads. A typical use case could be that some custom logic wrapped around Airflow tasks (e.g. config setups) needs to be executed before business functionality is run in a workload. In this example, we will create a custom arithmetic operator that will do some arithmetic operations as a part of our pipelineInstructionsThere are 2 tasks in this example, in part 1 we will understand how to create custom operators and upload them to a PyPI mirror. In part 2 we will use these operators in CDE.Pre-requisites :Create an account at https://test.pypi.org/Create and use API tokens to authenticate and upload your package. Refer to pypi.org documentation on using API tokens hereClone / Download the source code from the link herePart 1: Creating a Python package with a new Custom Airflow OperatorClone this repository in your local folderChange the current directory to the folder cde_custom_airflowoper

In this article, we will guide you through detailed, step-by-step instructions on how administrators can create a custom runtime image for notebooks in Cloudera AI, complete with custom extensions for VsCode.We'll also provide a fully functional runtime image as an example, which you can integrate seamlessly into your CML environment.Advantages: The following steps enable administrators to tailor a VsCode notebook by incorporating all the necessary extensions for end-users.Disadvantages: End-users are unable to permanently install extensions directly within a session; this must be done through the custom runtime image. Consequently, while users can add extensions during an active session, these will be removed once the session ends. However, the extensions included in the runtime image will persist.Custom docker runtime image: rcicakcloudera/vscodeextensions:latest Everything done below is emulated in the custom image above.Step 1: Write DockerfileBelow, you'll notice the ins

Hi cloudera I'm engineer of Kolon Benite that is cloudera partner in koreawhen I was other company, I made MyCloudera ID Please change MyCloudera account's ID and Company E-Mail addressPlease contact to me PhoneNumber)Thank you.

CDP Data Engineering (CDE) is the only cloud-native service purpose-built for enterprise data engineering teams. Building on Apache Spark, Data Engineering is an all-inclusive data engineering toolset that enables orchestration automation with Apache Airflow, advanced pipeline monitoring, visual troubleshooting, and comprehensive management tools to streamline ETL processes across enterprise analytics teams. Apache Iceberg is an open table format for huge analytic datasets. It provides features that, coupled with Spark as the compute engine, allow you to build data processing pipelines with dramatic gains in terms of scalability, performance, and overall developer productivity. CDE Provides native Iceberg support. With the release of CDE 1.20, the Spark Runtime has been updated with Apache Iceberg 1.3. This version introduces new features that provide great benefits to Data Engineers.  This article will familiarize you with Iceberg Table Branching and Tagging in CDE. Table Branch

CDP Data Engineering (CDE) is the only cloud-native service purpose-built for enterprise data engineering teams. Building on Apache Spark, Data Engineering is an all-inclusive data engineering toolset that enables orchestration automation with Apache Airflow, advanced pipeline monitoring, visual troubleshooting, and comprehensive management tools to streamline ETL processes across enterprise analytics teams. To manage job dependencies, CDE supports creating custom Python environments dedicated to Airflow using the airflow-python-env resource type. With this option, you can install custom libraries for running your Directed Acyclic Graphs (DAGs). The supported version is Python 3.8. A resource is a named collection of files or other resources referenced by a job. The airflow-python-env resource type allows you to specify a requirements.txt file that defines an environment that you can then activate globally for airflow deployments in a virtual cluster. You can install and use custom Pyt

CDP Data Engineering (CDE) is the only cloud-native service purpose-built for enterprise data engineering teams. Building on Apache Spark, Data Engineering is an all-inclusive data engineering toolset that enables orchestration automation with Apache Airflow, advanced pipeline monitoring, visual troubleshooting, and comprehensive management tools to streamline ETL processes across enterprise analytics teams. Git repositories allow teams to collaborate, manage project artifacts, and promote applications from lower to higher environments. CDE supports integration with Git providers such as GitHub, GitLab, and Bitbucket to synchronize job runs with different versions of your code. CDEPY is a package that allows you to do all the above with the convenience of Python. With it, you can remotely connect to a Virtual Cluster from your local machine or 3rd party tool as long as it supports Python. In this tutorial, you will use CDEPY to create a CDE Repository from a Git repository and create a

Extracting NiFi Provenance Data using SiteToSiteProvenanceReportingTask Part 1 In this tutorial, we will learn how configure NiFi to send provenance data to a second NiFi instance: Downloading NiFi Configure Site-To-Site Setting up the first Flow Setting up the Provenance Reporting Instance Flow Adding Site To Site Provenance Reporting Starting the flow Inspecting the Provenance data Next Steps NOTE: In this tutorial we are going to be taking the following shortcuts, in the spirit of understanding the concepts. Specifically we are going to: Run two NiFi instances on the same host. Not configure security in either NiFi instance or use a secure transport between NiFi hosts. These are not the best practices that would be recommended in a production environment.References For a primer on NiFi please refer to the NiFi Getting Started Guide.Downloading and Configuring NiFi Downlaod the latest version of NiFi from here Extract the contents of the download, we will refer to

Background: This is a new article from the Partner Solutions Engineering Team describing how to connect to CDW Hive using ODBC on your Mac. In this series, we will explore client-side activities and Partner integrations that may sometimes be atypical. Our goal is to offer convenient approaches that reduce time and research on your part. Step 0: Ensure that homebrew and keytool are installed on your Mac and in your path.% export PATH=$PATH:/opt/homebrew/bin% which keytool/opt/homebrew/opt/openjdk/bin/keytool Step 1: Download the CDW certificates for the Hive virtual cluster you want to connect to. Assuming you downloaded the certificate chain into a file called truststore.jks (can be any name you want), create a PEM file:% keytool -list -rfc -keystore truststore.jks | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > cdw-certs.pemYou should see multiple certificates in this file. Please also note that the /etc/hosts file on your ODBC client should be able to resolve the virtu

Purpose: Run SELECT to ingest data from Oracle 19c, and save the data into Azure ADLS Gen2 object storage, in Parquet format. Steps Step 1 Prepare the environment Make sure the Oracle 19c environment works well.  Prepare an Oracle table:    CREATE TABLE demo_sample ( column1 NUMBER, column2 NUMBER, column3 NUMBER, column4 VARCHAR2(10), column5 VARCHAR2(10), column6 VARCHAR2(10), column7 VARCHAR2(10), column8 VARCHAR2(10), column9 VARCHAR2(10), column10 VARCHAR2(10), column11 VARCHAR2(10), column12 VARCHAR2(10), CONSTRAINT pk_demo_sample PRIMARY KEY (column1, column2, column3, column4, column5, column6, column7, column8, column9));   Prepare 20

IntroductionIn this previous article, we talked about the benefits of deploying COD on Cloud Storage, covering the challenges involved with maintaining the consistency of HBase data and the different attempts to solve that problem, with a detailed technical description of the Store File Tracking (SFT) feature we've now been using for COD. Now we can successfully move our dataset to the cloud storage file system of preference, decoupling compute from storage and benefiting from optimal resource usage and independent scale. Moving the data away from RegionServers, however,  comes with a (performance) cost: the additional latency overhead of having to access the cloud storage. Our initial benchmarks showed that COD over cloud storage performed, on average, five times slower than COD clusters using HDFS on block storage with HDD (for a specific vendor).  HBase already ships built-in cache capabilities, including support for file-based caching. Also all major cloud provi

A Hive_on_Tez job goes through the following stages during its execution Query Submission When a user submits a Hive query, either through the Hive command line interface (CLI), HiveServer2, or through an application such as Hue, the query is sent to the Hive service for processing Query Parsing and Compilation The Hive service parses the submitted query to understand its structure and requirements. It then compiles the query into an execution plan specifying the steps needed to execute it. This execution plan includes details such as the sequence of Tez tasks required to perform the computation. Tez Session Initialization Once the optimized execution plan is ready, Hive initializes a Tez session. The Tez session provides a runtime environment for executing the query using Apache Tez, a framework optimized for processing large-scale data. This initialization process involves setting up necessary configurations, loading required libraries, and establishing communication channels with th

CDE is the Cloudera Data Engineering Service, a containerized managed service for the Cloudera Data Platform designed for Large Scale Batch and Streaming Pipelines with Spark, Airflow, and Iceberg. It allows you to submit batch jobs to auto-scaling virtual clusters. As a Cloud-Native service, CDE enables you to spend more time on your applications, and less time on infrastructure. A CDE Session is an interactive short-lived development environment for running Python, Scala, and Spark commands to help you iterate upon and build your Spark workloads. You can use CDE Sessions in CDE Virtual Clusters of type "All Purpose - Tier 2". A resource in CDE is a named collection of files used by a job or a session. Resources can include application code, configuration files, custom Docker images, and Python virtual environment specifications As of version 1.20, CDE introduces the ability to leverage CDE Resources when running CDE Interactive Sessions. This means that you can leverage fil

Introduction Cloudera Operational Database (COD) serves as a foundational service within the Cloudera Data Platform (CDP), enabling users to effortlessly create operational databases that dynamically scale to meet workload demands. When deploying high-performance applications at scale, a robust operational database plays a crucial role. COD addresses this need by offering a highly scalable and high-performance operational database engineered to support data-intensive applications. Leveraging the robust foundations of Apache HBase and Apache Phoenix, COD is integrated into the Cloudera Data Platform (CDP) in the public cloud. It is designed for versatility, accommodating hybrid as well as multi-cloud deployments, ensuring adaptability across various cloud environments including Amazon AWS, Microsoft Azure, and Google GCP. For AWS deployments, COD provides two primary storage options: S3 with Ephemeral Cache, known for its high performance albeit with slightly higher costs, and S3 withou

We will be using the Ozone ldb CLI tool to view the RocksDb table, which stores all the information in various tables. Here I will be showing how to check the table data that it has created for the filesystem in Ozone. [root@ccycloud-1 data]# pwd/var/lib/hadoop-ozone/scm/data[root@ccycloud-1 data]# pwd/var/lib/hadoop-ozone/om/data We need to traverse to the Ozone metadata directory through Cloudera Manager UI > configuration and search for "metadata". [root@ccycloud-1 data]# ozone debug ldb --db=om.db lsdefaultfileTableprincipalToAccessIdsTabledeletedTableuserTables3SecretTabletransactionInfoTableopenKeyTablesnapshotInfoTabledirectoryTableprefixTablemultipartInfoTablevolumeTabletenantStateTabledeletedDirectoryTabletenantAccessIdTableopenFileTablesnapshotRenamedTabledTokenTablemetaTablekeyTablebucketTable  Under the directory /var/lib/hadoop-ozone/scm/data we need need to ls to list out the tables.[root@ccycloud-1 data]# ozone debug ldb --db=scm.db lsde

Introduction Cloudera Operational Database (COD) is a service that runs on the Cloudera Data Platform (CDP). COD enables you to create a new operational database that automatically scales based on your workload. To deploy high-performance applications at scale, a rugged operational database is essential. COD is a high-performance and highly scalable operational database designed for powering, at any scale, the biggest data applications on the planet. Powered by Apache HBase and Apache Phoenix, COD ships out of the box with Cloudera Data Platform (CDP) in the public cloud. It’s also ready for hybrid and multi-cloud deployments to meet your business where it is today, whether AWS, Microsoft Azure, or GCP. Support for cloud storage is an important capability of COD that, in addition to the pre-existing support for HDFS on local storage, offers customers a choice of price-performance characteristics. Please refer to the blog for more information on the performance differences between COD o

IntroductionApache HBase provides different cache capabilities. In Cloudera Operational Database (COD) deployments using cloud storage, we leverage the file-based BucketCache implementation to deploy it over ephemeral SSD volumes to cache the whole user dataset to avoid client reads from having to reach the slower cloud storage layer. However, the file-based BucketCache implementation was originally volatile, meaning the cache was not retained upon region server restarts. This requires a warm-up period every time a region server is restarted, to maintain optimal read performance. Depending on the data size and the configured cache size, this warm-up can take anywhere from a few minutes to a few hours. To eliminate this, the Cloudera team and the HBase community implemented the BucketCache persistence feature (HBASE-27264/HBASE-27486), where the region servers periodically persist the blocks cached in the bucket cache. This persisted information is then used to recover the cache in the

Simple API to find region name for given row key. It will also give you hint about the region - Where key will be placed.To compile class:$JAVA_HOME/bin/javac -cp `hbase classpath`: Regionfinder.javaTo run tool:$JAVA_HOME/bin/java -cp `hbase classpath`: Regionfinder <tablename> <rowkey>Regionfinder.javaimport java.io.IOException;import org.apache.hadoop.conf.Configuration;import org.apache.hadoop.hbase.HBaseConfiguration;import org.apache.hadoop.hbase.HRegionLocation;import org.apache.hadoop.hbase.TableName;import org.apache.hadoop.hbase.client.Connection;import org.apache.hadoop.hbase.client.ConnectionFactory;import org.apache.hadoop.hbase.client.Get;import org.apache.hadoop.hbase.client.RegionLocator;import org.apache.hadoop.hbase.client.Result;import org.apache.hadoop.hbase.client.Table;import org.apache.hadoop.hbase.util.Bytes;public class Regionfinder { private static void usage(String[] args) { if (args.length < 2) { Sys

Earlier this year Cloudera AI added a new way to accelerate GenAI projects by tapping into Hugging Face Spaces and deploying these projects right inside of CML with just a few clicks. With over 6,500 spaces as of this writing, Hugging Face community is still growing rapidly and provides a convenient platform for practitioners and organizations to share their work in areas from classical machine learning to the latest GenAI research. In this article you will learn how to enable and use this feature to accelerate your own ML projects.The default Hugging Face Spaces AMP catalog is enabled for all CML Public Cloud workspaces starting from version 2.0.43-b208. To enable users to launch external Hugging Face AMPs, additional steps are necessary (see end of this article).Steps to Deploy Hugging Face Space AMPLet's dive right in and see how simple it is to deploy a Hugging Face AMP:Click on AMPs in the left sidebar of your ML Workspace. If you don’t see this, then AMPs are not enabled by your

Overview Once Data Engineers complete the development and testing of their workloads, moving to deployment can be a challenging task. This process often involves a lengthy checklist, including the modification of multiple configurations that differ across environments. For instance, the configuration spark.yarn.access.hadoopFileSystems must now point to the Data Lake or Data Store appropriate for the production environment. Data Engineers may also opt to use a "staging" environment to test job performance, necessitating changes to these configurations twice: initially from development to staging, and subsequently for production deployment. A recommended practice in such cases is to use templating the configurations within the workload. This approach allows for easier management of deployments across different environments. In this article, we demonstrate a pattern of templating Data Engineering workloads in Cloudera Data Engineering in Airflow. This best practice allows workload deploy

Introduction Apache Hadoop's efficiency and reliability significantly depend on the Java Virtual Machine (JVM)'s performance. To assist in monitoring and analyzing JVM pauses, especially those induced by Garbage Collection (GC), Hadoop integrates a utility named JvmPauseMonitor, a Hadoop common class introduced by https://issues.apache.org/jira/browse/HADOOP-9618 and widely used in Hadoop projects like HDFS, Hbase, Yarn, Oozie, MapReduce, etc. Overview of JvmPauseMonitor Link to reference code here Key Features and Implementation     private String formatMessage(long extraSleepTime, Map<String, GcTimes> gcTimesAfterSleep, Map<String, GcTimes> gcTimesBeforeSleep) { Set<String> gcBeanNames = Sets.intersection( gcTimesAfterSleep.keySet(), gcTimesBeforeSleep.keySet()); List<String> gcDiffs = Lists.newArrayList(); for (String name : gcBeanNames) { GcTimes diff = gcTimesAfterSleep.get(name).subtract( gcTimes

This article explains how to configure Spark Connections in Cloudera Machine Learning. CML enables easy Spark data connections to the data stored in the Data Lake by abstracting the SparkSession connection details. CML users don't need to set complicated endpoint and configuration parameters or load HWC or Iceberg libraries. They can use the cml.data library to get preconfigured connections. CML Data Connection Snippets  The spark SparkSession object has all the necessary options set to make connections work. Users have two options to set custom or job-specific configurations like executor CPU and memory parameters. 1. Specify the configuration inlineWith SparkContext's setSystemProperty method, you can set Spark properties that will be picked up while building the SparkSession object. You can set any Spark property like below before calling cmldata's get_spark_session method.   Python script import cml.data_v1 as cmldatafrom pyspa