Community Articles

 SummaryThis article details the resolution of a startup failure for Knox Gateway roles during a Data Hub upgrade, involving the reconfiguration of a missing knox_pac4j_password.SymptomsWhile upgrading the Data Hub cluster, Knox Gateway services failed to start.Error message: java.lang.IllegalArgumentException: The variable [${knox_pac4j_password}] does not have a corresponding value.The upgrade process was halted due to this failure.CauseThe probable cause of the startup failure was an improperly generated or missing knox_pac4j_password during the upgrade process.The built-in upgrade handler should have automatically generated the Knox secret, but it did not.InstructionsTo resolve the issue, the following steps were taken:Verify if an entry for knox_pac4j_password exists in the Cloudera Manager database.Execute SQL query: SELECT * FROM configs WHERE attr LIKE '%knox_pac4j_password%';Generate a 16-bit UUID random password.Use OpenSSL: openssl ran

SummaryResolved service errors on Datanode#09 by identifying and addressing NFS mount issues. The Cloudera agent was reconfigured to bypass NFS checks temporarily, allowing services to return to a healthy state.SymptomsMultiple service errors reported on Datanode#09.Cloudera agent was not in contact with Cloudera Manager.Filesystem usage for certain nodev filesystems was unknown due to an inactive worker process.CauseThe NFS partition was not properly mounted on the host, which led to problems with the Cloudera agent's health checks.The Cloudera agent configuration was set to monitor NFS mounts (monitored_nodev_filesystem_types=nfs,nfs4,tmpfs), which failed due to the unmounted NFS partition.InstructionsTemporarily comment out the NFS monitoring line in the Cloudera agent configuration file to bypass the check and restore agent communication:Open /etc/cloudera-scm-agent/config.ini using a text editor.Comment out the line monitored_nodev_filesystem_types=nfs,nfs4,tmpfs.Re

NiFi's content repository will hold on to claims until there are no active FlowFiles still anywhere on the canvas referencing that claim. This can result in very old or very large claims being left in the content repo using up space. Something as small as a single 0 byte FlowFile that sitting in some queue may end up preventing a multi-gigabyte content claim from being removed. The intent of this article is help users understand how to find those active FlowFiles and clear them form your dataflow.-There is no simple UI feature or NiFi rest-api endpoint users can use to return information linking FlowFiles to exiting content claims; however, all is not lost. Users can add an additional indexed field to your provenance configuration to so that is starts indexing the "ContentClaimIdentifier" associated with each FlowFile event generated. This would give you the ability to use Provenance to identify all the FlowFiles associated to specific claim in the content repo.-To add this, simply a

Servers used in this document:node1.localdomain = HAProxy servernode2.localdomain = NameNode1node3.localdomain = NameNode21. Install HAProxy[root@node1 ~]# yum install -y haproxy2. Set up HAProxy with minimum configuration[root@node1 ~]# cp -p /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.orig[root@node1 ~]# vim /etc/haproxy/haproxy.cfg... (snip) ...#---------------------------------------------------------------------# main frontend which proxys to the backends#---------------------------------------------------------------------frontend main *:50070 default_backend app#---------------------------------------------------------------------# round robin balancing between the various backends#---------------------------------------------------------------------backend app balance roundrobin option httpchk GET /webhdfs/v1/?op=CHECKACCESS http-check expect rstatus ([23][0-9][0-9]|401) server node2 node2.localdomain:50070 check server node3 node3.localdomain:50070 chec

It's early 2025 and agentic workflows are the hottest commodities within AI. Giving large language models (LLMs) the ability to use tools to complete actions opens up a huge range of use cases. At the same time, it's important to ensure that the AI agents we build are secure.Why AI Agents?AI Agents have the potential to transform society and how we leverage technology to complete routine tasks. Some key use cases for AI Agents can involve:Automation Customer Service TasksOrchestrating Business Processes & WorkflowsTaking Actions Based on Unstructured DataIn this article you will get familiar the CrewAI library and how we can build fully private AI agents with CrewAI and Cloudera's AI Inference Service. Powered by Nvidia NeMo Inference Microservices (NIM), Cloudera's AI Inference Service gives you the ability to deploy private LLMs within your infrastructure. This article is designed to walk you through the setup and application of these tools in an agentic worfklow context to creat

Prerequisites: This article assumes some knowledge of Spark and Iceberg. These examples make use of Cloudera Data Engineering and Cloudera Data Warehouse.Note: Iceberg branch-based WAP requires Spark 3.5 or above. Motivation (simulated scenario) Let's imagine we have a high-traffic table many users use in the platform. Each day, before the start of a work day, a Spark job populates this table with new records. On most days, this simple architecture will work quite well. However, this morning the Spark job had a data error and wrote out a batch of duplicate records. This data quality issue was reported by one of the users and the data engineers are now trying to fix it. However, during this time, access to the high-traffic table is obviously disrupted. Of course, data quality checks can be implemented to catch issues like this earlier. However, this still leaves you little time to roll back changes or fix the data issues before users are affected. Write Audit Publish a pattern These t

JupyterLab is a powerful, web-based interactive development environment widely used for data science, machine learning, and Spark application development. It extends the classic Jupyter Notebook interface, offering a more flexible and integrated workspace.For Spark application development, JupyterLab provides the following advantages:1. Interactive Exploration: Run Spark queries and visualize results interactively, making it ideal for data exploration and debugging.2. Rich Visualization: Seamlessly integrate with Python visualization libraries like Matplotlib, Seaborn, and Plotly to analyze and interpret Spark data.3. Ease of Integration: Use PySpark or Sparkmagic to connect JupyterLab with Spark clusters, enabling distributed data processing from the notebook environment.Spark Connect is a feature introduced in Apache Spark that provides a standardized, client-server architecture for connecting to Spark clusters. It decouples the client from the Spark runtime, allowing users to intera

PyCharm is a popular integrated development environment (IDE) for Python, developed by JetBrains. It provides a comprehensive set of tools designed to boost productivity and simplify the coding process. When working with Apache Spark, PyCharm can be an excellent choice for managing Python-based Spark applications. Its features include:Code Editing and Debugging: PyCharm offers intelligent code completion, syntax highlighting, and robust debugging tools that simplify Spark application development.Virtual Environments and Dependency Management: PyCharm makes it easy to configure Python environments with Spark libraries and manage dependencies.Notebook Support: With built-in support for Jupyter Notebooks, PyCharm allows you to work interactively with data, making it easier to visualize and debug Spark pipelines.Version Control: PyCharm integrates with Git and other version control systems, simplifying collaboration and project management.Spark Connect is a feature introduced in Apach

Spark GraphFrames is a package within Apache Spark that allows users to perform graph processing operations on data using a DataFrame-based approach. It will enable you to perform various graph operations like finding connected components, calculating shortest paths, identifying triangles, and more. Real world applications include social network analysis, recommendation systems, web link structures, flight connections, and other scenarios where relationships between data points are crucial. Cloudera AI (CAI) is a cloud-native service within the Cloudera that enables enterprise data science teams to collaborate across the full data lifecycle. It provides immediate access to enterprise data pipelines, scalable compute resources, and preferred tools, streamlining the process of moving analytic workloads from research to production. Using Spark GraphFrames in Cloudera AI requires minimum configuration effort. This quickstart provides a basic example so you can get started quickly. Requirem

To Fix under-replicated blocks in HDFS, below is quick instruction to use:####Fix under-replicated blocks### su - <$hdfs_user>bash-4.1$ hdfs fsck / | grep 'Under replicated' | awk -F':' '{print $1}' >> /tmp/under_replicated_files -bash-4.1$ for hdfsfile in `cat /tmp/under_replicated_files`; do echo "Fixing $hdfsfile :" ; hadoop fs -setrep 3 $hdfsfile; done

Problem Statement: Model governanceData science and model building are prevalent activities that bring new and innovative value to enterprises. The more prevalent this activity becomes, the more problematic model governance becomes. Model governance typically centers on these questions: What models were deployed? when? where? What was the serialized object deployed?Was deployment to a microservice, a Spark context, other?What version was deployed?How do I trace the deployed model to its concrete details: the code, its training data, owner, Read.me overview, etc?Apache Atlas is the central tool in organizing, searching and accessing metadata of data assets and processes on your Hadoop platform. Its Rest API can push metadata from anywhere, so Atlas can also represent metadata off your Hadoop cluster.Atlas lets you define your own types of objects and inherit from existing out-of-the box types. This lets you store whatever metadata you want to store, and to tie this into Atlas's pow

Why this article is created?The main reason for creating and listing all the legacy configurations for Spark 3 is to ease the migration from Spark 2 to Spark 3.Main concern here was that during migration Spark 3 behaves different than Spark 2 due to upstream changes and causing the same code to work differently.Legacy configuration in Spark 3Property nameDefaultMeaningspark.sql.catalog.your_catalog_name.pushDownAggregatespark.sql.catalog.your_catalog_name.pushDownLimitspark.sql.catalog.your_catalog_name.pushDownOffsetTrueSince Spark 3.5, the JDBC options related to DS V2 pushdown are true by default. These options include: pushDownAggregate, pushDownLimit, pushDownOffset and pushDownTableSample.To restore the legacy behavior, please set them to false. e.g. set spark.sql.catalog.your_catalog_name.pushDownAggregate to false.spark.sql.legacy.negativeIndexInArrayInsertFalseSince Spark 3.5, the array_insert function is 1-based for negative indexes. It inserts new element at the end of input

Spark Python Integration Test Result Exceptions   In this article, just I talk about exceptions and their Python and Spark versions. Keep on watching this article where I will add some more exceptions and solutions. 1. TypeError: an integer is required (got type bytes) Exception: Traceback (most recent call last): File "/opt/pyspark_udf_example.py", line 3, in <module> from pyspark.sql import SparkSession File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 655, in _load_unlocked File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible File "<frozen zipimport>", line 259, in load_module File "/opt/spark/python/lib/pyspark.zip/pyspark/__init__.py", line 46, in <module> File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._b

IntroductionImpala is a horizontally scalable, distributed database engine that is best suited for fast BI workloads. Apache Iceberg is an open table format that has brought new and exciting capabilities to the big data world with support for deletes, updates, time travel, partition management, and more. With Iceberg, users can insert data into tables that have advanced and evolving partition layouts. Data warehouse users often need to comply with regulations like GDPR and CCPA, requiring them to delete records from their warehouses. They might also need to update their data from time to time. These operations often require regular removal of old partitions. Impala and Iceberg can easily handle these use cases. Frequent write operations can CREATE TABLE ice_tbl (i INT, s STRING, ts TIMESTAMP)STORED BY ICEBERG; potentially lead to performance issues, including the small file problem, or the overhead of handling lots of delete files. To resolve these issues, Impala can als

For users experiencing trouble uploading a file to a case using the standard ‘UPLOAD FILE(s)’ method from the Case Attachment tab, the following alternatives exist:Upload files from the command line30GB LimitThis method utilizes a java program to allow case attachments to be uploaded via the command line. This method is intended for those who are experiencing trouble with the case attachment UI or who prefer to upload a file from the command line.  This method also supports proxy parametersHow to use:Download the JARRun the program:Usage: java -jar cloudera.attachment.cli.jar [case_number] [email_address] [file_name...]Example:  java -jar cloudera.attachment.cli.jar 55555 customer@company.com logs1.log logs2.logThe above example uploads logs1.log and logs2.log to case number 55555 from customer@company.comSupported proxy command line properties include:  -Dhttp.proxyHost=, -Dhttp.proxyPort=, -Dhttp.proxyUsername=, -Dhttp.proxyPassword=Example: java -j

IntroductionLet's be clear up front: by "tactical", I mean, "not recommended for production use" 🙂 There are several better ways to achieve code re-use in Cloudera Data Engineering, including adding custom operators and libraries at the Virtual Cluster level, and @VishRajagopalan's CDE Dynamic Package Loader.However, if you just need something more "quick and dirty", then read on to learn how to load Python code from Resources in Cloudera Data Engineering.To quote the official documentation:A Resource in Cloudera Data Engineering is a named collection of files used by a job or a session. Resources can include application code, configuration files, custom Docker images, and Python virtual environment specifications (requirements.txt).The key to understanding how to leverage Resources to provide custom code to your Airflow Jobs in Cloudera Data Engineering is to know how exactly Resources are made available at run-time. Once you know that, the rest is just "normal" Python

Objective This article contains basic instructions for creating and deploying a Custom Runtime with VS Code and Nvidia libraries to your Cloudera AI Workspace. With this runtime, you can increase your productivity when developing your deep learning use cases in CML Sessions. This example is also valid for Cloudera Data Science Workbench (CDSW) clusters. If you are using CML or CDSW for deep learning, GenAI, or LLM use cases, please scroll down to the bottom of this page to check out more examples. Using Custom Runtimes Cloudera ML Runtimes are a set of Docker images created to enable machine learning development and host data applications in the Cloudera Data Platform (CDP) and the Cloudera AI service. ML Runtimes provide a flexible, fully customizable, lightweight development and production machine learning environment for both CPU and GPU processing frameworks while enabling unfettered access to data, on-demand resources, and the ability to install and use any libraries/algorithms wi

Cloudera Data Flow is an integrated data management platform designed to handle real-time streaming data pipelines. It allows organizations to ingest, process, analyze, and distribute data in motion across various environments such as on-premises, public clouds, or hybrid architectures.Cloudera CDF leverages Kubernetes as the foundational infrastructure for deploying, managing, and scaling data flow applications.Cloudera CDF also uses NiFi as the core engine for data movement, transformation, and management. When developing or troubleshooting DataFlows you might need to set processors on DEBUG mode.NiFi uses Logback as the default logging framework to handle logging of application activities  and error messages.With the below examples you can set a processor in to DEBUG mode by just knowing the processor name.The below example assumes that you are pointing to your appropriate kubeconfig file a an environment variable on your shell named KUBECONFIG.However

Table of Contents: Apache SolrSpark Solr ConnectorSpark Solr Collection IntroductionSpark Solr IntegrationTroubleshootingApache Solr1.1 Solr Introduction  Apache Solr (stands for Searching On Lucene w/ Replication) is the popular, blazing-fast, open-source enterprise search platform built on Apache Lucene. It is designed to provide powerful full-text search, faceted search, and indexing capabilities to enable fast and accurate search functionality for various types of data. Solr is the second-most popular enterprise search engine after Elasticsearch. Written in Java, Solr has RESTful XML/HTTP and JSON APIs and client libraries for many programming languages such as Java, Phyton, Ruby, C#, PHP, and many more being used to build search-based and big data analytics applications for websites, databases, files, etc. Solr is often used as a search engine for applications, websites, and enterprise systems that require robust search capabilities. It

In part I of this series, we reviewed preliminaries related to SSO, including LDAP authentication for Ambari, and we set up an application in Okta that would correspond to our KnoxSSO service provider for the SAML authentication flow. We are now ready to configure Knox within Ambari.We will replace the Form-based IdP configuration that Knox comes with out of the box with the pac4j federation provider. Pac4j is a Java security library and it is used as a federation provider within Knox to support the OAuth, CAS, SAML and OpenID Connect protocols. It must be used for SSO, in association with the KnoxSSO service (and optionally with the SSOCookieProvider for access to REST APIs).In Ambari, we'll navigate to Knox > Config > Advanced knoxsso-topology and add XML similar to the following:<topology> <gateway> <provider> <role>federation</role> <name>pac4j</name> <enabled>true</enabled> <param&

In today's data-driven world, smooth access to cloud platforms is crucial. But things can get messy with so many tools and logins to manage. This article dives into the power of integrating Google Workspace SSO with the Cloudera Data Platform, offering a seamless and secure user experience for your team. Cloudera: Laying the Groundwork Setting up SSO might involve a few administrative tasks within Cloudera. Specifically, you'll need to create an identity provider to capture the connection details for your Google Workspace account. But don't worry, this step only requires a Cloudera account administrator or PowerUser role. Let's explore how to navigate this process and unlock the benefits of seamless SSO. Here's how to get started (Note: Requires Account Administrator or PowerUser privileges): Head to the Cloudera Console: Sign in and navigate to the Cloudera home page. Click on "Management Console" from there. Identity Provider Setup: In the User Management section of the side navigat

ObjectiveThis article provides an introduction to the Iceberg using Spark SQL in Cloudera Data Engineering. CDE provides native Apache Iceberg Table Format support in its Spark Runtimes. This means you can create and interact with Iceberg Table format tables without any configurations.AbstractCDP Data Engineering (CDE) is the only cloud-native service purpose-built for enterprise data engineering teams. Building on Apache Spark, Data Engineering is an all-inclusive data engineering toolset that enables orchestration automation with Apache Airflow, advanced pipeline monitoring, visual troubleshooting, and comprehensive management tools to streamline ETL processes across enterprise analytics teams.Apache Iceberg is an open table format format for huge analytic datasets. It provides features that, coupled with Spark as the compute engine, allows you to build data processing pipelines with dramatic gains in terms of scalability, performance, and overall developer productivity.Iceberg is na

The limitation of display of 5000 tables is due to dbms.py script under Hue, below is the snippet of the script:def get_tables(self, database='default', table_names='*'):hql = "SHOW TABLES IN %s '%s'" % (database, table_names) # self.client.get_tables(database, table_names) is too slowquery = hql_query(hql)handle = self.execute_and_wait(query, timeout_sec=15.0)if handle:result = self.fetch(handle, rows=5000)self.close(handle)return [name for table in result.rows() for name in table]else:return []To increase the number of the tables displayed in Hue. Please do the following:1. cp /usr/lib/hue/apps/beeswax/src/beeswax/server/dbms.py /tmp 2. Stop Hue service as 'service hue stop'. 3. Edit /usr/lib/hue/apps/beeswax/src/beeswax/server/dbms.py to change the rows value to 8000.if handle:result = self.fetch(handle, rows=8000) --> This value needs to be changed.self.close(handle)return [name for table in result.rows() for name in table]else:return [] 4. Restart hue as 'service hue restart'.

Table of ContentsOverviewDesignDesign ExplanationImplementationPrerequisitesStep #1 - Setup Cloudera Data FlowStep #2 - Setup Cloudera Data WarehouseStep #3 - ExecuteOverviewCloudera and IBM have partnered to create industry-leading, enterprise-grade data and AI services using open-source ecosystems—all designed to achieve faster data and analytics at scale.This article shows an end-to-end flow to process real-time unstructured data with GenAI using Cloudera's DataFlow and Data Warehouse, and IBM's watsonx.ai.DesignDesign ExplanationBased on the incoming documents in AWS S3 bucket, NiFi prepares the input for all the watsonx.ai models.NiFi calls watsonx.ai model (granite-13b-instruct-v1) to Extract the key fields in the document. See the sample IBM watsonx.ai prompt below. NiFi calls watsonx.ai model (granite-13b-chat-v1) to Summarize the information in the document. See the sample IBM watsonx.ai prompt below. NiFi calls watsonx.ai model (granite-13b-instruct-v1) to

IntroductionCloudera Operational Database is a service that runs on the Cloudera. Cloudera Operational Database enables you to create a new operational database that automatically scales based on your workload.In this blog, we’ll discuss a new autoscaling mechanism in Cloudera Operational Database that is designed to address the computing needs of workloads dynamically in the fastest possible manner. This new mechanism is most beneficial for HDFS storage-based clusters, but is also useful for non-ephemeral cache-based clusters using cloud storage.Note: This type of auto-scaling is not supported for clusters using large SSD-based cache with cloud storage.BackgroundAutoscaling is an important aspect of cloud computing that helps organizations manage their computing resources effectively and ensures cost efficiency. It enables organizations to deal with unpredictable workloads and provides the advantages of improved resource utilization and cost-effectiveness.In Cloudera Operational Datab