Resources

Objective This article contains basic instructions for creating and deploying a Custom Runtime with VS Code and Nvidia libraries to your Cloudera AI Workspace. With this runtime, you can increase your productivity when developing your deep learning use cases in CML Sessions. This example is also valid for Cloudera Data Science Workbench (CDSW) clusters. If you are using CML or CDSW for deep learning, GenAI, or LLM use cases, please scroll down to the bottom of this page to check out more examples. Using Custom Runtimes Cloudera ML Runtimes are a set of Docker images created to enable machine learning development and host data applications in the Cloudera Data Platform (CDP) and the Cloudera AI service. ML Runtimes provide a flexible, fully customizable, lightweight development and production machine learning environment for both CPU and GPU processing frameworks while enabling unfettered access to data, on-demand resources, and the ability to install and use any libraries/algorithms wi

Please reply to this topic with a short introduction of yourself. Include details like your background, what industry and job role you are in, what projects you are planning, and a fun fact about you.We encourage all our members to participate and connect.

Cloudera Data Flow is an integrated data management platform designed to handle real-time streaming data pipelines. It allows organizations to ingest, process, analyze, and distribute data in motion across various environments such as on-premises, public clouds, or hybrid architectures.Cloudera CDF leverages Kubernetes as the foundational infrastructure for deploying, managing, and scaling data flow applications.Cloudera CDF also uses NiFi as the core engine for data movement, transformation, and management. When developing or troubleshooting DataFlows you might need to set processors on DEBUG mode.NiFi uses Logback as the default logging framework to handle logging of application activities  and error messages.With the below examples you can set a processor in to DEBUG mode by just knowing the processor name.The below example assumes that you are pointing to your appropriate kubeconfig file a an environment variable on your shell named KUBECONFIG.However

Table of Contents: Apache SolrSpark Solr ConnectorSpark Solr Collection IntroductionSpark Solr IntegrationTroubleshootingApache Solr1.1 Solr Introduction  Apache Solr (stands for Searching On Lucene w/ Replication) is the popular, blazing-fast, open-source enterprise search platform built on Apache Lucene. It is designed to provide powerful full-text search, faceted search, and indexing capabilities to enable fast and accurate search functionality for various types of data. Solr is the second-most popular enterprise search engine after Elasticsearch. Written in Java, Solr has RESTful XML/HTTP and JSON APIs and client libraries for many programming languages such as Java, Phyton, Ruby, C#, PHP, and many more being used to build search-based and big data analytics applications for websites, databases, files, etc. Solr is often used as a search engine for applications, websites, and enterprise systems that require robust search capabilities. It

In part I of this series, we reviewed preliminaries related to SSO, including LDAP authentication for Ambari, and we set up an application in Okta that would correspond to our KnoxSSO service provider for the SAML authentication flow. We are now ready to configure Knox within Ambari.We will replace the Form-based IdP configuration that Knox comes with out of the box with the pac4j federation provider. Pac4j is a Java security library and it is used as a federation provider within Knox to support the OAuth, CAS, SAML and OpenID Connect protocols. It must be used for SSO, in association with the KnoxSSO service (and optionally with the SSOCookieProvider for access to REST APIs).In Ambari, we'll navigate to Knox > Config > Advanced knoxsso-topology and add XML similar to the following:<topology> <gateway> <provider> <role>federation</role> <name>pac4j</name> <enabled>true</enabled> <param&

In today's data-driven world, smooth access to cloud platforms is crucial. But things can get messy with so many tools and logins to manage. This article dives into the power of integrating Google Workspace SSO with the Cloudera Data Platform, offering a seamless and secure user experience for your team. Cloudera: Laying the Groundwork Setting up SSO might involve a few administrative tasks within Cloudera. Specifically, you'll need to create an identity provider to capture the connection details for your Google Workspace account. But don't worry, this step only requires a Cloudera account administrator or PowerUser role. Let's explore how to navigate this process and unlock the benefits of seamless SSO. Here's how to get started (Note: Requires Account Administrator or PowerUser privileges): Head to the Cloudera Console: Sign in and navigate to the Cloudera home page. Click on "Management Console" from there. Identity Provider Setup: In the User Management section of the side navigat

ObjectiveThis article provides an introduction to the Iceberg using Spark SQL in Cloudera Data Engineering. CDE provides native Apache Iceberg Table Format support in its Spark Runtimes. This means you can create and interact with Iceberg Table format tables without any configurations.AbstractCDP Data Engineering (CDE) is the only cloud-native service purpose-built for enterprise data engineering teams. Building on Apache Spark, Data Engineering is an all-inclusive data engineering toolset that enables orchestration automation with Apache Airflow, advanced pipeline monitoring, visual troubleshooting, and comprehensive management tools to streamline ETL processes across enterprise analytics teams.Apache Iceberg is an open table format format for huge analytic datasets. It provides features that, coupled with Spark as the compute engine, allows you to build data processing pipelines with dramatic gains in terms of scalability, performance, and overall developer productivity.Iceberg is na

The limitation of display of 5000 tables is due to dbms.py script under Hue, below is the snippet of the script:def get_tables(self, database='default', table_names='*'):hql = "SHOW TABLES IN %s '%s'" % (database, table_names) # self.client.get_tables(database, table_names) is too slowquery = hql_query(hql)handle = self.execute_and_wait(query, timeout_sec=15.0)if handle:result = self.fetch(handle, rows=5000)self.close(handle)return [name for table in result.rows() for name in table]else:return []To increase the number of the tables displayed in Hue. Please do the following:1. cp /usr/lib/hue/apps/beeswax/src/beeswax/server/dbms.py /tmp 2. Stop Hue service as 'service hue stop'. 3. Edit /usr/lib/hue/apps/beeswax/src/beeswax/server/dbms.py to change the rows value to 8000.if handle:result = self.fetch(handle, rows=8000) --> This value needs to be changed.self.close(handle)return [name for table in result.rows() for name in table]else:return [] 4. Restart hue as 'service hue restart'.

Table of ContentsOverviewDesignDesign ExplanationImplementationPrerequisitesStep #1 - Setup Cloudera Data FlowStep #2 - Setup Cloudera Data WarehouseStep #3 - ExecuteOverviewCloudera and IBM have partnered to create industry-leading, enterprise-grade data and AI services using open-source ecosystems—all designed to achieve faster data and analytics at scale.This article shows an end-to-end flow to process real-time unstructured data with GenAI using Cloudera's DataFlow and Data Warehouse, and IBM's watsonx.ai.DesignDesign ExplanationBased on the incoming documents in AWS S3 bucket, NiFi prepares the input for all the watsonx.ai models.NiFi calls watsonx.ai model (granite-13b-instruct-v1) to Extract the key fields in the document. See the sample IBM watsonx.ai prompt below. NiFi calls watsonx.ai model (granite-13b-chat-v1) to Summarize the information in the document. See the sample IBM watsonx.ai prompt below. NiFi calls watsonx.ai model (granite-13b-instruct-v1) to

IntroductionCloudera Operational Database is a service that runs on the Cloudera. Cloudera Operational Database enables you to create a new operational database that automatically scales based on your workload.In this blog, we’ll discuss a new autoscaling mechanism in Cloudera Operational Database that is designed to address the computing needs of workloads dynamically in the fastest possible manner. This new mechanism is most beneficial for HDFS storage-based clusters, but is also useful for non-ephemeral cache-based clusters using cloud storage.Note: This type of auto-scaling is not supported for clusters using large SSD-based cache with cloud storage.BackgroundAutoscaling is an important aspect of cloud computing that helps organizations manage their computing resources effectively and ensures cost efficiency. It enables organizations to deal with unpredictable workloads and provides the advantages of improved resource utilization and cost-effectiveness.In Cloudera Operational Datab

SummaryPre-requisitesHow-To-do in Power BI DesktopStep1 : Install the [Impala ODBC Connector 2.7.0 for Cloudera Enterprise]Step 2: Copy the ODBC folder to Power BI Desktop folderStep 3: Edit the config file to specify ODBC DriverStep 4: Edit the config file to specify ODBC Driver parameterHow-To-do in Power BI Service (On-premise Data Gateway)Step 1: Edit the config file to specify DriverStep 2: Edit the Simba .ini file to specify Driver parameterSummaryLast week I posted an article [How to Connect to Impala Using the Power BI Desktop + Cloudera ODBC Impala Driver with Kerberos Authentication],So far (2024 Sep),  CDP Public Cloud CDW supports Basic Authentication (HTTP), today I will share how to Connect to CDW Impala VW Using the Power BI Desktop + Cloudera ODBC Impala Driver with Basic Authentication. Pre-requisitesPower BI Desktop Edition https://www.microsoft.com/en-us/power-platform/products/power-bi/desktopImpala in CDP Public Cloud CDWImpala ODBC Connector 2.7.0 f

Objective Cloudera Data Engineering is a cloud-native service provided by Cloudera. It is designed to simplify and enhance the development, deployment, and management of data engineering workloads at scale. CDE is part of the Cloudera Data Platform (CDP), which is a comprehensive, enterprise-grade platform for managing and analyzing data across hybrid and multi-cloud environments. Cloudera Data Engineering offers several advantages. With CDE, you can create a "CDE Spark-Submit" using the same syntax as your regular Spark-Submit. Alternatively, you can specify your Spark-Submit as a "CDE Job of type Spark" using a reusable Job Definition, which enhances observability, troubleshooting, and dependency management. These unique capabilities of CDE are especially useful for Spark Data Engineers who develop and deploy Spark Pipelines at scale. This includes working with different Spark-Submit definitions and dynamic, complex dependencies across multiple clusters. For example, when packaging a

Background Apache ZooKeeper is a core infrastructure component in the Apache Hadoop stack and is widely used by many companies for service discovery, configuration management, etc. In a previous blog post, we described improving ZooKeeper security by enabling SASL Quorum Peer Mutual Authentication and Authorization in Cloudera Distribution of Hadoop (CDH). That was back in 2017 and since then the open source community has released Apache ZooKeeper 3.8.0 with important security improvements and new features. It’s the latest stable version included in the Cloudera Data Platform today.  In this blog post, we’ll focus on wire encryption, specifically how to enable it and how Cloudera Manager helps secure all communication channels between ZooKeeper clients and servers. The previous blog post covered authentication and authorization of quorum members which is a huge improvement in terms of security, but to close gaps entirely and avoid eavesdropping on your important data, encryption i

Objective Cloudera AI is a platform designed to help organizations build, deploy, and manage machine learning models at scale. It is part of Cloudera’s suite of enterprise data platforms and solutions, focusing on providing a robust environment for data scientists, analysts, and engineers to collaborate on end-to-end machine learning workflows. PyGurobi is a Python interface for the Gurobi Optimizer, a powerful and widely used solver for mathematical optimization problems. Gurobi is known for its high performance in solving a variety of optimization problems, including linear programming (LP), quadratic programming (QP), mixed-integer programming (MIP), and others. In this tutorial, you will use PyGurobi on CML to optimize product prices and maximize enterprise revenue. Requirements The following are required to reproduce this example: CML Workspace in AWS, Azure, OCP, or ECS. Basic knowledge of Python for Machine Learning including Sci-Kit Learn, Spark, Iceberg, and XGBoost. You shou

ObjectiveCloudera Data Engineering is a cloud native service for Cloudera Data Platform that allows you to submit batch jobs to auto-scaling virtual clusters. CDE enables you to spend more time on your applications, and less time on infrastructure.Wheels allow for faster installations and more stability in the package distribution process. In the context of PySpark, Wheels allow you to make python dependent modules available to executors without having to do pip install dependencies on every node and to use application source code as a package.In this tutorial you will create a CDE Spark Job using a Wheel file via the CDE CLI.Project RequirementsIn order to execute the Hands On Labs you need:* A Spark 3 and Iceberg-enabled CDE Virtual Cluster (Azure, AWS and Private Cloud ok).* The CDE CLI installed on your local machine. If you need to install it for the first time please follow these steps.* Familiarity with Python, PySpark and the CDE CLI is highly recommended.* No script code chang

Currently, there is no option in Atlas to delete classifications from deleted entities through the UI. When an entity is deleted, its classification may still appear, but there is no direct method to remove it via the Atlas interface.Workaround:To delete a classification from a deleted entity, you can use the following API command: curl -iv -u {username} -X DELETE https://{atlashost:port}/api/atlas/v2/entity/guid/{guid}/classification/{Classification Name}Example: If:Username: adminAtlas Host: cloudera.comPort: 21000GUID: 95cdef42-8586-4933-b7b4-XXXXXXXXXClassification Name: Test1Then, the command will be:curl -iv -u admin -X DELETE https://cloudera.com:21000/api/atlas/v2/entity/guid/95cdef42-8586-4933-b7b4-d4942cbd5afb/classification/Test1The expected response should be:HTTP/1.1 204 No ContentYou can find the GUID of historical entities from the following resources:=>https://atlas.apache.org/api/v2/resource_EntityREST.html#resource_EntityREST_deleteClassificationByUniqueAttrib

Spark Python Supportability MatrixThe Spark Python Supportability Matrix serves as an essential tool for determining which Python versions are compatible with specific Spark versions. This matrix provides a detailed overview of the compatibility levels for various Python versions across different Spark releases.Spark VersionPython Min Supported VersionPython Max Supported VersionPythonv 2.7Pythonv3.4Pythonv3.5Pythonv 3.6Pythonv3.7Pythonv3.8Pythonv3.9Pythonv3.10Pythonv3.113.5.23.83.11NoNoNoNoNoYesYesYesYes3.5.13.83.11NoNoNoNoNoYesYesYesYes3.5.03.83.11NoNoNoNoNoYesYesYesYes3.4.23.73.11NoNoNoNoYesYesYesYesYes3.4.13.73.11NoNoNoNoYesYesYesYesYes3.4.03.73.11NoNoNoNoYesYesYesYesYes3.3.33.73.10NoNoNoNoYesYesYesYesNo3.3.23.73.10NoNoNoNoYesYesYesYesNo3.3.13.73.10NoNoNoNoYesYesYesYesNo3.3.03.73.10NoNoNoNoYesYesYesYesNo3.2.43.63.9NoNoNoYesYesYesYesNoNo3.2.33.63.9NoNoNoYesYesYesYesNoNo3.2.23.63.9NoNoNoYesYesYesYesNoNo3.2.13.63.9NoNoNoYesYesYesYesNoNo3.2.03.63.9NoNoNoYesYesYesYesNoNo3.1.33.63.9NoNoN

Until now, you could query JDBC data sources from Hive using JDBC Storage Handler. However, with the release of CDP 7.2.18 (Public Cloud runtime) and 7.1.9 SP 1 (Private Cloud Base), you can also query Hive tables hosted in a different environment using a similar method. Here's a brief demo:   0: jdbc:hive2://ccycloud-1.sbiswal.root.como> CREATE EXTERNAL TABLE loc_hive_table(c1 int, c2 string) STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler' TBLPROPERTIES(. . . . . . . . . . . . . . . . . . . . . . .> "hive.sql.database.type"="HIVE",. . . . . . . . . . . . . . . . . . . . . . .> "hive.sql.jdbc.driver"="org.apache.hive.jdbc.HiveDriver",. . . . . . . . . . . . . . . . . . . . . . .> "hive.sql.jdbc.url"="jdbc:hive2://sbiswal-jdbctest-master0.sup-defa.xxxxxxx.a4.cloudera.site/db1;ssl=true;transportMode=http;httpPath=sbiswal-jdbctest/cdp-proxy-api/hive",. . . . . . . . . . . . . . . . . . . . . . .> "hive.sql.dbcp.username”= ”user1”,. . . . . . . . . . . . . .

In this article, I'll walk you through a Flink application I developed to process real-time data and write the output to HDFS within the Cloudera Data Platform (CDP). But first, let’s discuss why Flink is a powerful choice for streaming analytics and how Cloudera’s platform can help you make the most of it.Why Flink?Flink excels in scenarios where low-latency processing and real-time analytics are critical. Compared to Spark, Flink often shines in streaming use cases due to its advanced event-time processing and lower latency. However, Spark remains a robust choice for batch processing and when integrating with existing Spark-based pipelines.This flexibility is where Cloudera stands out as the obvious choice for streaming analytics. Cloudera supports both Flink and Spark, giving you the power to choose the right tool for your specific use case. Beyond just tooling, Cloudera’s hybrid platform also allows you to reduce your cloud bill by running applications on-premise, while maintaining

1. Introduction2. Executor memory3. Memory Management4. Static Memory Manager (SMM)5. Unified Memory Manager (UMM)6. Understand the Memory Allocation using Spark UI 1. Introduction Spark is an in-memory processing engine where all of the computation that a task does happens in memory. So, it is important to understand Spark Memory Management. This will help us develop Spark applications and perform performance tuning. If the memory allocation is too large when committing, it will occupy resources. If the memory allocation is too small, memory overflow and full-GC problems will occur easily. Efficient memory use is critical for good performance, but the reverse is also true: inefficient memory use leads to bad performance. Spark Architecture:   The Spark application includes two JVM processes: driver and executor. The driver is the main control process, which is responsible for creating the SparkSession/SparkContext, submitting th

At times in a CDH/CDP cluster, users may have multiple HS2 mapped under a load balancer to achieve high availability and user connection load balancing. However, for scenarios like separating high-priority jobs, dedicated queue/HS2 requirement for a business unit, dedicated resource allocation, etc., users may have a dedicated HS2 in their cluster.  Below are two methods to create and connect to a dedicated Hive HS2 using ZooKeeper details: Method 1: Connect using HS2 instance hostname/IP. Add an instance with an HS2 role on it. Ensure that this HS2 is not part of your load balancer, in case you want to keep it separate for certain jobs. But this is optional. To connect to this dedicated HS2 use the below beeline command:beeline> !connect jdbc:hive2://<serverName>:10000/default;​     Method 2: Connect using Zookeeper instance details. Go to Hive_on Tez service > Instances. Add a new HS2 instance. Click on the "Role Groups" option on the top right side

This guide provides a comprehensive overview of using JOLT within Apache NiFi. It is designed to help developers and data engineers efficiently transform JSON data, offering clear instructions and practical examples.This is an unofficial resource, created out of a personal initiative to fill the knowledge gap.The guide includes: Introduction to JOLT JOLT (JSON-to-JSON Transformation Language) enables flexible and powerful JSON data transformations. It allows you to restructure, filter, and modify JSON objects to fit specific requirements. Getting Started Begin with the JoltTransformJSON processor in NiFi. The guide provides setup instructions and tips for using the JOLT editor window to streamline your transformation tasks. Using the JOLT Editor The JOLT editor in NiFi includes features for creating and debugging transformation specifications. Our guide explains how to use these features effectively. Writing JOLT Specifications We explain the structure of JOLT specifications, focusing

Problem Overview As organizations mature in implementing AI systems, they are looking to standardize workflows across different machine learning use cases. Here is one key challenge to address in the ML lifecycle. ML teams spend considerable time in the development lifecycle on data exploration and data analysis activities. These tasks are crucial because the type and quality of the datasets used as inputs for model training directly affect how much time is spent training the model and how well it performs afterward. Before being used for model training, attributes in a dataset known as features go through a lot of changes. These transformations can be simple, e.g., changing a categorical variable such as “opt for email subscriptions” from yes and no to boolean values 1 and 0. Or, it can be complex, including merging multiple input fields into a new field, e.g., labeling a song with a combination of attributes such as music genre, decade of origin, country, and band size to come u

What is Content Repository Archiving?There are three properties in the nifi.properties file that deal with the archiving on content in the NiFi Content Repository.The default NiFi values for these are shown below:nifi.content.repository.archive.max.retention.period=12 hoursnifi.content.repository.archive.max.usage.percentage=50%nifi.content.repository.archive.enabled=trueThe purpose of content archiving is so that users can view and/ or replay content via the provenance UI that is no longer in their dataflow(s). The configured values do not have any impact on the amount of provenance history that is retained. If content associated to a particular provenance event no longer exists in the content archive, provenance will simply report to the user that the content is not available.The content archive is kept in within the same directory or directories where you have configured your content repository(s) to exist. When a "content claim" is archived, that claim is moved in to an archive sub

Cloudera recently  announced the release of Kubernetes Operators for Data in Motion. These Kubernetes Operators  enable customers to deploy Apache NiFi, Apache Kafka, and Apache Flink clusters on Kubernetes application platforms such as Red Hat OpenShift, the industry’s leading hybrid cloud application platform powered by Kubernetes. With these Kubernetes Operators, customers can easily deploy end-to-end data streaming capabilities on their existing Kubernetes clusters and benefit from auto-scaling, efficient resource management, and streamlined setup and operations. In this blog post, we will revisit the advantages of running Apache NiFi on Kubernetes and take a closer look at the architecture and deployment model when using the Cloudera Flow Management Kubernetes Operator.Advantages of running Apache NiFi on KubernetesApache NiFi is a powerful tool for building data movement pipelines using a visual flow designer. Hundreds of built-in processors make it easy to connect