Community Articles

SummaryPre-requisitesHow-To-do in Power BI DesktopStep1 : Install the [Impala ODBC Connector 2.7.0 for Cloudera Enterprise]Step 2: Copy the ODBC folder to Power BI Desktop folderStep 3: Edit the config file to specify ODBC DriverStep 4: Edit the config file to specify ODBC Driver parameterHow-To-do in Power BI Service (On-premise Data Gateway)Step 1: Edit the config file to specify DriverStep 2: Edit the Simba .ini file to specify Driver parameterSummaryLast week I posted an article [How to Connect to Impala Using the Power BI Desktop + Cloudera ODBC Impala Driver with Kerberos Authentication],So far (2024 Sep),  CDP Public Cloud CDW supports Basic Authentication (HTTP), today I will share how to Connect to CDW Impala VW Using the Power BI Desktop + Cloudera ODBC Impala Driver with Basic Authentication. Pre-requisitesPower BI Desktop Edition https://www.microsoft.com/en-us/power-platform/products/power-bi/desktopImpala in CDP Public Cloud CDWImpala ODBC Connector 2.7.0 f

Objective Cloudera Data Engineering is a cloud-native service provided by Cloudera. It is designed to simplify and enhance the development, deployment, and management of data engineering workloads at scale. CDE is part of the Cloudera Data Platform (CDP), which is a comprehensive, enterprise-grade platform for managing and analyzing data across hybrid and multi-cloud environments. Cloudera Data Engineering offers several advantages. With CDE, you can create a "CDE Spark-Submit" using the same syntax as your regular Spark-Submit. Alternatively, you can specify your Spark-Submit as a "CDE Job of type Spark" using a reusable Job Definition, which enhances observability, troubleshooting, and dependency management. These unique capabilities of CDE are especially useful for Spark Data Engineers who develop and deploy Spark Pipelines at scale. This includes working with different Spark-Submit definitions and dynamic, complex dependencies across multiple clusters. For example, when packaging a

Background Apache ZooKeeper is a core infrastructure component in the Apache Hadoop stack and is widely used by many companies for service discovery, configuration management, etc. In a previous blog post, we described improving ZooKeeper security by enabling SASL Quorum Peer Mutual Authentication and Authorization in Cloudera Distribution of Hadoop (CDH). That was back in 2017 and since then the open source community has released Apache ZooKeeper 3.8.0 with important security improvements and new features. It’s the latest stable version included in the Cloudera Data Platform today.  In this blog post, we’ll focus on wire encryption, specifically how to enable it and how Cloudera Manager helps secure all communication channels between ZooKeeper clients and servers. The previous blog post covered authentication and authorization of quorum members which is a huge improvement in terms of security, but to close gaps entirely and avoid eavesdropping on your important data, encryption i

Objective Cloudera AI is a platform designed to help organizations build, deploy, and manage machine learning models at scale. It is part of Cloudera’s suite of enterprise data platforms and solutions, focusing on providing a robust environment for data scientists, analysts, and engineers to collaborate on end-to-end machine learning workflows. PyGurobi is a Python interface for the Gurobi Optimizer, a powerful and widely used solver for mathematical optimization problems. Gurobi is known for its high performance in solving a variety of optimization problems, including linear programming (LP), quadratic programming (QP), mixed-integer programming (MIP), and others. In this tutorial, you will use PyGurobi on CML to optimize product prices and maximize enterprise revenue. Requirements The following are required to reproduce this example: CML Workspace in AWS, Azure, OCP, or ECS. Basic knowledge of Python for Machine Learning including Sci-Kit Learn, Spark, Iceberg, and XGBoost. You shou

ObjectiveCloudera Data Engineering is a cloud native service for Cloudera Data Platform that allows you to submit batch jobs to auto-scaling virtual clusters. CDE enables you to spend more time on your applications, and less time on infrastructure.Wheels allow for faster installations and more stability in the package distribution process. In the context of PySpark, Wheels allow you to make python dependent modules available to executors without having to do pip install dependencies on every node and to use application source code as a package.In this tutorial you will create a CDE Spark Job using a Wheel file via the CDE CLI.Project RequirementsIn order to execute the Hands On Labs you need:* A Spark 3 and Iceberg-enabled CDE Virtual Cluster (Azure, AWS and Private Cloud ok).* The CDE CLI installed on your local machine. If you need to install it for the first time please follow these steps.* Familiarity with Python, PySpark and the CDE CLI is highly recommended.* No script code chang

Currently, there is no option in Atlas to delete classifications from deleted entities through the UI. When an entity is deleted, its classification may still appear, but there is no direct method to remove it via the Atlas interface.Workaround:To delete a classification from a deleted entity, you can use the following API command: curl -iv -u {username} -X DELETE https://{atlashost:port}/api/atlas/v2/entity/guid/{guid}/classification/{Classification Name}Example: If:Username: adminAtlas Host: cloudera.comPort: 21000GUID: 95cdef42-8586-4933-b7b4-XXXXXXXXXClassification Name: Test1Then, the command will be:curl -iv -u admin -X DELETE https://cloudera.com:21000/api/atlas/v2/entity/guid/95cdef42-8586-4933-b7b4-d4942cbd5afb/classification/Test1The expected response should be:HTTP/1.1 204 No ContentYou can find the GUID of historical entities from the following resources:=>https://atlas.apache.org/api/v2/resource_EntityREST.html#resource_EntityREST_deleteClassificationByUniqueAttrib

Spark Python Supportability MatrixThe Spark Python Supportability Matrix serves as an essential tool for determining which Python versions are compatible with specific Spark versions. This matrix provides a detailed overview of the compatibility levels for various Python versions across different Spark releases.Spark VersionPython Min Supported VersionPython Max Supported VersionPythonv 2.7Pythonv3.4Pythonv3.5Pythonv 3.6Pythonv3.7Pythonv3.8Pythonv3.9Pythonv3.10Pythonv3.113.5.23.83.11NoNoNoNoNoYesYesYesYes3.5.13.83.11NoNoNoNoNoYesYesYesYes3.5.03.83.11NoNoNoNoNoYesYesYesYes3.4.23.73.11NoNoNoNoYesYesYesYesYes3.4.13.73.11NoNoNoNoYesYesYesYesYes3.4.03.73.11NoNoNoNoYesYesYesYesYes3.3.33.73.10NoNoNoNoYesYesYesYesNo3.3.23.73.10NoNoNoNoYesYesYesYesNo3.3.13.73.10NoNoNoNoYesYesYesYesNo3.3.03.73.10NoNoNoNoYesYesYesYesNo3.2.43.63.9NoNoNoYesYesYesYesNoNo3.2.33.63.9NoNoNoYesYesYesYesNoNo3.2.23.63.9NoNoNoYesYesYesYesNoNo3.2.13.63.9NoNoNoYesYesYesYesNoNo3.2.03.63.9NoNoNoYesYesYesYesNoNo3.1.33.63.9NoNoN

Until now, you could query JDBC data sources from Hive using JDBC Storage Handler. However, with the release of CDP 7.2.18 (Public Cloud runtime) and 7.1.9 SP 1 (Private Cloud Base), you can also query Hive tables hosted in a different environment using a similar method. Here's a brief demo:   0: jdbc:hive2://ccycloud-1.sbiswal.root.como> CREATE EXTERNAL TABLE loc_hive_table(c1 int, c2 string) STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler' TBLPROPERTIES(. . . . . . . . . . . . . . . . . . . . . . .> "hive.sql.database.type"="HIVE",. . . . . . . . . . . . . . . . . . . . . . .> "hive.sql.jdbc.driver"="org.apache.hive.jdbc.HiveDriver",. . . . . . . . . . . . . . . . . . . . . . .> "hive.sql.jdbc.url"="jdbc:hive2://sbiswal-jdbctest-master0.sup-defa.xxxxxxx.a4.cloudera.site/db1;ssl=true;transportMode=http;httpPath=sbiswal-jdbctest/cdp-proxy-api/hive",. . . . . . . . . . . . . . . . . . . . . . .> "hive.sql.dbcp.username”= ”user1”,. . . . . . . . . . . . . .

In this article, I'll walk you through a Flink application I developed to process real-time data and write the output to HDFS within the Cloudera Data Platform (CDP). But first, let’s discuss why Flink is a powerful choice for streaming analytics and how Cloudera’s platform can help you make the most of it.Why Flink?Flink excels in scenarios where low-latency processing and real-time analytics are critical. Compared to Spark, Flink often shines in streaming use cases due to its advanced event-time processing and lower latency. However, Spark remains a robust choice for batch processing and when integrating with existing Spark-based pipelines.This flexibility is where Cloudera stands out as the obvious choice for streaming analytics. Cloudera supports both Flink and Spark, giving you the power to choose the right tool for your specific use case. Beyond just tooling, Cloudera’s hybrid platform also allows you to reduce your cloud bill by running applications on-premise, while maintaining

1. Introduction2. Executor memory3. Memory Management4. Static Memory Manager (SMM)5. Unified Memory Manager (UMM)6. Understand the Memory Allocation using Spark UI 1. Introduction Spark is an in-memory processing engine where all of the computation that a task does happens in memory. So, it is important to understand Spark Memory Management. This will help us develop Spark applications and perform performance tuning. If the memory allocation is too large when committing, it will occupy resources. If the memory allocation is too small, memory overflow and full-GC problems will occur easily. Efficient memory use is critical for good performance, but the reverse is also true: inefficient memory use leads to bad performance. Spark Architecture:   The Spark application includes two JVM processes: driver and executor. The driver is the main control process, which is responsible for creating the SparkSession/SparkContext, submitting th

At times in a CDH/CDP cluster, users may have multiple HS2 mapped under a load balancer to achieve high availability and user connection load balancing. However, for scenarios like separating high-priority jobs, dedicated queue/HS2 requirement for a business unit, dedicated resource allocation, etc., users may have a dedicated HS2 in their cluster.  Below are two methods to create and connect to a dedicated Hive HS2 using ZooKeeper details: Method 1: Connect using HS2 instance hostname/IP. Add an instance with an HS2 role on it. Ensure that this HS2 is not part of your load balancer, in case you want to keep it separate for certain jobs. But this is optional. To connect to this dedicated HS2 use the below beeline command:beeline> !connect jdbc:hive2://<serverName>:10000/default;​     Method 2: Connect using Zookeeper instance details. Go to Hive_on Tez service > Instances. Add a new HS2 instance. Click on the "Role Groups" option on the top right side

This guide provides a comprehensive overview of using JOLT within Apache NiFi. It is designed to help developers and data engineers efficiently transform JSON data, offering clear instructions and practical examples.This is an unofficial resource, created out of a personal initiative to fill the knowledge gap.The guide includes: Introduction to JOLT JOLT (JSON-to-JSON Transformation Language) enables flexible and powerful JSON data transformations. It allows you to restructure, filter, and modify JSON objects to fit specific requirements. Getting Started Begin with the JoltTransformJSON processor in NiFi. The guide provides setup instructions and tips for using the JOLT editor window to streamline your transformation tasks. Using the JOLT Editor The JOLT editor in NiFi includes features for creating and debugging transformation specifications. Our guide explains how to use these features effectively. Writing JOLT Specifications We explain the structure of JOLT specifications, focusing

Problem Overview As organizations mature in implementing AI systems, they are looking to standardize workflows across different machine learning use cases. Here is one key challenge to address in the ML lifecycle. ML teams spend considerable time in the development lifecycle on data exploration and data analysis activities. These tasks are crucial because the type and quality of the datasets used as inputs for model training directly affect how much time is spent training the model and how well it performs afterward. Before being used for model training, attributes in a dataset known as features go through a lot of changes. These transformations can be simple, e.g., changing a categorical variable such as “opt for email subscriptions” from yes and no to boolean values 1 and 0. Or, it can be complex, including merging multiple input fields into a new field, e.g., labeling a song with a combination of attributes such as music genre, decade of origin, country, and band size to come u

What is Content Repository Archiving?There are three properties in the nifi.properties file that deal with the archiving on content in the NiFi Content Repository.The default NiFi values for these are shown below:nifi.content.repository.archive.max.retention.period=12 hoursnifi.content.repository.archive.max.usage.percentage=50%nifi.content.repository.archive.enabled=trueThe purpose of content archiving is so that users can view and/ or replay content via the provenance UI that is no longer in their dataflow(s). The configured values do not have any impact on the amount of provenance history that is retained. If content associated to a particular provenance event no longer exists in the content archive, provenance will simply report to the user that the content is not available.The content archive is kept in within the same directory or directories where you have configured your content repository(s) to exist. When a "content claim" is archived, that claim is moved in to an archive sub

Cloudera recently  announced the release of Kubernetes Operators for Data in Motion. These Kubernetes Operators  enable customers to deploy Apache NiFi, Apache Kafka, and Apache Flink clusters on Kubernetes application platforms such as Red Hat OpenShift, the industry’s leading hybrid cloud application platform powered by Kubernetes. With these Kubernetes Operators, customers can easily deploy end-to-end data streaming capabilities on their existing Kubernetes clusters and benefit from auto-scaling, efficient resource management, and streamlined setup and operations. In this blog post, we will revisit the advantages of running Apache NiFi on Kubernetes and take a closer look at the architecture and deployment model when using the Cloudera Flow Management Kubernetes Operator.Advantages of running Apache NiFi on KubernetesApache NiFi is a powerful tool for building data movement pipelines using a visual flow designer. Hundreds of built-in processors make it easy to connect

Each rule in the auth-to-local rules as the following formats:RULE:[n:string](regexp)s/pattern/replacement/RULE:[n:string](regexp)s/pattern/replacement/gRULE:[n:string](regexp)s/pattern/replacement//LRULE:[n:string](regexp)s/pattern/replacement/g/L [n:string]Indicates a matching rule where n declares the number of expected components in the principal. Components are separated by a /, where a user account has one component (ambari-qa) and a service account has two components (nn/fqdn). The string value declares how to reformat the value to be used in the rest of the expression. The placeholders are as follows:$0 - realm$1 - 1st component$2 - 2nd componentTypically we ignore the 2nd component since it is the service’s hostname and thus the format is generally set to $1@$0 (but can be any pattern) as in:[1:$1@$0]Matches on ambari-qa@EXAMPLE.COMTranslates to ambari-qa@EXAMPLE.COM[2:$1@$0]Matches on nn/c6501.ambari.apache.org@EXAMPLE.COMTranslates to nn@EXAMPLE.COM (regexp)Indicates a

Prerequisite :1. We should have the fsimage to continue with this analysis.2.Access to git What is small files ?A small file is one which is significantly smaller than the default Apache Hadoop HDFS default block size (128MB by default in CDH/CDP). One should note that it is expected and inevitable to have some small files on HDFS. These are files like library jars, XML configuration files, temporary staging files, and so on. But when small files become a significant part of datasets, the problems arise.Problems with small files and HDFSSmall files are a common challenge in the Apache Hadoop world and when not handled with care, they can lead to a number of complications. The Apache Hadoop Distributed File System (HDFS) was developed to store and process large data sets over the range of terabytes and petabytes. However, HDFS stores small files inefficiently, leading to inefficient Namenode memory utilisation and RPC calls, block scanning throughput degradation, and reduced applic

IntroductionApache Iceberg HistoryApache Spark and Iceberg Supportability Matrix TableReferences Introduction The Spark and Iceberg Supportability Matrix provides comprehensive information regarding the compatibility and supportability of Spark and Iceberg versions with various operating systems, frameworks, and dependencies. Apache Iceberg History The development of Iceberg was started in 2017 by Netflix. The project was open-sourced and donated to the Apache Software Foundation in November 2018. In May 2020, the Iceberg project graduated to become a top-level Apache project. Apache Iceberg 0.7.0 was released on Oct 26, 2019 (Incubating) Apache Iceberg 0.8.0 was released on May 7, 2020 (Incubating). Apache Iceberg 0.9.0 was released on Jul 14, 2020. Apache Iceberg 0.9.1 was released on Aug 11, 2020. Apache Iceberg 0.10.0 was released on Nov 12, 2020. Apache Iceberg 0.11.0 was released on Jan 27, 2021. Apache Iceberg 0.11.1 was released on Apr 3, 2021. Apache Iceberg 0

  In this article, I will document how to use CFM 1.0.1.0 to interact with Apache Impala. This article still applies if using HDF / Apache NiFi   The latest official JDBC driver that will work when using NiFi is the  JDBC driver 2.6.4 or below. At the time of this writing, any driver above that causes class conflicts with the NiFi JVM and the driver's own use of log4j.   Pre-requisite   Downloading and extracting the JDBC driver:   The JDBC drivers can be found at Impala JDBC Connector 2.6.15 for Cloudera Enterprise. Select 2.6.4 * or if in the future a version higher than 2.6.15 is available, use that. Download and extract 2.6.4 and make note of where it extracts to. Ensure that the user that runs the NiFi JVM ( nifi ) has permission to that path. The jar file that you will use is called ImpalaJDBC41.jar.   Create Impala table and load dataset sample to HDFS:   Use this data set tips.csv and add it to your HDFS. hdfs dfs

PurposeBuilding Custom Operators helps Data engineers abstract and reuse code across workloads. A typical use case could be that some custom logic wrapped around Airflow tasks (e.g. config setups) needs to be executed before business functionality is run in a workload. In this example, we will create a custom arithmetic operator that will do some arithmetic operations as a part of our pipelineInstructionsThere are 2 tasks in this example, in part 1 we will understand how to create custom operators and upload them to a PyPI mirror. In part 2 we will use these operators in CDE.Pre-requisites :Create an account at https://test.pypi.org/Create and use API tokens to authenticate and upload your package. Refer to pypi.org documentation on using API tokens hereClone / Download the source code from the link herePart 1: Creating a Python package with a new Custom Airflow OperatorClone this repository in your local folderChange the current directory to the folder cde_custom_airflowoper

In this article, we will guide you through detailed, step-by-step instructions on how administrators can create a custom runtime image for notebooks in Cloudera AI, complete with custom extensions for VsCode.We'll also provide a fully functional runtime image as an example, which you can integrate seamlessly into your CML environment.Advantages: The following steps enable administrators to tailor a VsCode notebook by incorporating all the necessary extensions for end-users.Disadvantages: End-users are unable to permanently install extensions directly within a session; this must be done through the custom runtime image. Consequently, while users can add extensions during an active session, these will be removed once the session ends. However, the extensions included in the runtime image will persist.Custom docker runtime image: rcicakcloudera/vscodeextensions:latest Everything done below is emulated in the custom image above.Step 1: Write DockerfileBelow, you'll notice the ins

CDP Data Engineering (CDE) is the only cloud-native service purpose-built for enterprise data engineering teams. Building on Apache Spark, Data Engineering is an all-inclusive data engineering toolset that enables orchestration automation with Apache Airflow, advanced pipeline monitoring, visual troubleshooting, and comprehensive management tools to streamline ETL processes across enterprise analytics teams. Apache Iceberg is an open table format for huge analytic datasets. It provides features that, coupled with Spark as the compute engine, allow you to build data processing pipelines with dramatic gains in terms of scalability, performance, and overall developer productivity. CDE Provides native Iceberg support. With the release of CDE 1.20, the Spark Runtime has been updated with Apache Iceberg 1.3. This version introduces new features that provide great benefits to Data Engineers.  This article will familiarize you with Iceberg Table Branching and Tagging in CDE. Table Branch

CDP Data Engineering (CDE) is the only cloud-native service purpose-built for enterprise data engineering teams. Building on Apache Spark, Data Engineering is an all-inclusive data engineering toolset that enables orchestration automation with Apache Airflow, advanced pipeline monitoring, visual troubleshooting, and comprehensive management tools to streamline ETL processes across enterprise analytics teams. To manage job dependencies, CDE supports creating custom Python environments dedicated to Airflow using the airflow-python-env resource type. With this option, you can install custom libraries for running your Directed Acyclic Graphs (DAGs). The supported version is Python 3.8. A resource is a named collection of files or other resources referenced by a job. The airflow-python-env resource type allows you to specify a requirements.txt file that defines an environment that you can then activate globally for airflow deployments in a virtual cluster. You can install and use custom Pyt

CDP Data Engineering (CDE) is the only cloud-native service purpose-built for enterprise data engineering teams. Building on Apache Spark, Data Engineering is an all-inclusive data engineering toolset that enables orchestration automation with Apache Airflow, advanced pipeline monitoring, visual troubleshooting, and comprehensive management tools to streamline ETL processes across enterprise analytics teams. Git repositories allow teams to collaborate, manage project artifacts, and promote applications from lower to higher environments. CDE supports integration with Git providers such as GitHub, GitLab, and Bitbucket to synchronize job runs with different versions of your code. CDEPY is a package that allows you to do all the above with the convenience of Python. With it, you can remotely connect to a Virtual Cluster from your local machine or 3rd party tool as long as it supports Python. In this tutorial, you will use CDEPY to create a CDE Repository from a Git repository and create a

Extracting NiFi Provenance Data using SiteToSiteProvenanceReportingTask Part 1 In this tutorial, we will learn how configure NiFi to send provenance data to a second NiFi instance: Downloading NiFi Configure Site-To-Site Setting up the first Flow Setting up the Provenance Reporting Instance Flow Adding Site To Site Provenance Reporting Starting the flow Inspecting the Provenance data Next Steps NOTE: In this tutorial we are going to be taking the following shortcuts, in the spirit of understanding the concepts. Specifically we are going to: Run two NiFi instances on the same host. Not configure security in either NiFi instance or use a secure transport between NiFi hosts. These are not the best practices that would be recommended in a production environment.References For a primer on NiFi please refer to the NiFi Getting Started Guide.Downloading and Configuring NiFi Downlaod the latest version of NiFi from here Extract the contents of the download, we will refer to