Resources

While running Hive query from Hue, the query fails with the following error:WARN [HiveServer2-Handler-Pool: Thread-41]: thrift.ThriftCLIService (ThriftCLIService.java:FetchResults(596)) - Error fetching results: org.apache.hive.service.cli.HiveSQLException: Couldn't find log associated with operation handle:OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=1cd26ef5-00f8-4003-be5c-65e2a542ee18] at org.apache.hive.service.cli.operation.OperationManager.getOperationLogRowSet (OperationManager.java:257) at org.apache.hive.service.cli.session.HiveSessionImpl.fetchResults(HiveSessionImpl.java:656) at sun.reflect.GeneratedMethodAccessor104.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:79) at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionPro

JDBC/ODBC Drivers for Hive can be downloaded from Cloudera website.The first thing we need to collect is the Hive Endpoint from Cloudera Management console. This can be found at the bottom of in the specific DataHub window. It will be in the following format: jdbc:hive2://datahub1-master0.geo-1035.lskx-pvue.a4.cloudera.site/;ssl=true;transportMode=http;httpPath=datahub1/cdp-proxy-api/hive Majorly we need to furnish the following information in the appropriate fields:Host(s) : datahub1-master0.geo-1035.lskx-pvue.a4.cloudera.sitePort : 443Authentication Mechanism : Username/PasswordThrift Transport : HTTPGo to HTTP Options :  HTTP Path : datahub1/cdp-proxy-api/hive  (you will get this info from the Hive Endpoint)Go to SSL Options:6.1. Check Enable SSL6.2. Check Allow Self-signed Server Certificate check box6.3. Trusted Certificates: Select the path to the PEM file containing the root ca cert of the Knox gateway.Note: You can download

Log Forwarding/Ingestion PatternsLog forwarding & ingestion is a key starting point for many logging initiatives such as log analytics, cyber security, anomaly & bot detection, etc etc. This article will focus few (not comprehensive) patterns for log forwarding/ingestion using NiFi. Commonly rsyslog is used to capture and ship log messages.“Rsyslog is an open-source software utility used on UNIX and Unix-like computer systems for forwarding log messages in an IPnetwork. It implements the basic syslog protocol, extends it with content-based filtering, rich filtering capabilities, flexible configuration options and adds features such as using TCP for transport.”More on how to configure rsyslog: hereNiFi is able to ingest messages from rsyslog over TCP or UDP via ListenSysLog processor. This allows for little to no coding. Patterns Pattern A A minimalist design. Rsyslog is configured to simply forward log messages to a NiFi cluster. Rsyslog /etc/rsyslog.conf file needs to be conf

Cloudera Data EngineeringCDE is the Cloudera Data Engineering Service, a containerized managed service for Cloudera Data Platform designed for Large Scale Batch Pipelines with Spark, Airflow and Iceberg. It allows you to submit batch jobs to auto-scaling virtual clusters. As a Cloud-Native service, CDE enables you to spend more time on your applications, and less time on infrastructure.CDE allows you to create, manage, and schedule Apache Spark jobs without the overhead of creating and maintaining Spark clusters. With CDE, you define virtual clusters with a range of CPU and memory resources, and the cluster scales up and down as needed to run your Spark workloads, helping to control your cloud costs.Cloudera Data Engineering provides a command line interface (CLI) client. You can use the CLI to create and update jobs, view job details, manage job resources, run jobs, and more.Apache IcebergApache Iceberg is a cloud-native, high-performance open table format for organizing petabyte-scal

Installing Spark3 and Livy3 on Cloudera Manager with CDS3 ParcelApache Spark Logo 1. IntroductionThis guide outlines the steps for installing Apache Spark 3.x on your Cloudera cluster by leveraging Cloudera Manager (CM) and CDS3 parcels. Learn how to efficiently download, distribute, and activate the CDS3 parcel for a seamless Spark3 deployment, saving you time and effort compared to traditional methods. Additionally, the guide provides resources for troubleshooting any potential issues encountered during the installation.Note: This article, mainly focuses on Cloudera Manager Server does have Internet access.2. CDP-compatible CDS3 Parcels DetailsCDS 3 Powered by Apache Spark is an add-on service for CDP Private Cloud Base, distributed as a parcel. The Cloudera Service Descriptor(CDS) file is available in Cloudera Manager for CDP 7.1.X.The CDS version label is constructed in v.v.v.w.w.xxxx.y-z...z format and carries the following information:v.v.v - Apache Spark upst

In this article, I will demonstrate how to access the Ozone file system using Java API. I  explain the steps to validate the Ozone cluster for volume, bucket, and key creation. Let's begin and create a Java project with Maven build:  Create a Java project with the name AccessOFS using IntelliJ from File->New->Project.The project structure will look like the below image: Modify your pom.xml and add the dependencies for ozone-client. The pom.xml will be as follows.<?xml version="1.0" encoding="UTF-8"?><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>org.demo</groupId> <artifactId>AccessOFS</artifactId> <version>1.0-SNAPSHOT</version> <build> <plugins> <

In this article I am going to review the required steps and processes to setup some NiFi SSL Context Services with modern versions of NiFi (1.20, 1.21, 2.0).    In the past, nifi installations did not come installed with SSL enabled.   If you were a NiFi admin and had to setup ssl, you know it was not an easy task, and could often times prevented the cluster from even being secured at all.   Thanks to the wonderful team of innovators working on the Apache NiFi Project, I am happy to show the Easy Button works to install a fully secured Nifi.  I will also show how to setup SSL Context Services for internal and external connection to https enabled endpoints. First, let's do a new install of Apache NiFi 1.21.   This article is not about how to do that, so fast forward to a running NiFi,  lets take a look at some important details that you will find in the nifi-app.log: Your login details:2023-06-01 10:02:55,493 INFO [main] o.a.n.

Logging Iceberg Metrics with MLFlow Tracking in CMLCloudera AI on Cloudera Data Platform (CDP) accelerates time-to-value by enabling data scientists to collaborate in a single unified platform that is all inclusive for powering any AI use case. Purpose-built for agile experimentation and production ML workflows, Cloudera Machine Learning manages everything from data preparation to MLOps, to predictive reporting.CML is compatible with the MLflow Tracking API and makes use of the MLflow client library as the default method to log experiments. Existing projects with existing experiments are still available and usable. CML’s experiment tracking features allow you to use the MLflow client library for logging parameters, code versions, metrics, and output files when running your machine learning code.Apache Iceberg is a table format for huge analytics datasets in the cloud that defines how metadata is stored and data files are organized. Iceberg is also a library that compute engines can use

CML Model Deployment with MLFlow and APIv2Cloudera AI on Cloudera Data Platform (CDP) accelerates time-to-value by enabling data scientists to collaborate in a single unified platform that is all inclusive for powering any AI use case. Purpose-built for agile experimentation and production ML workflows, Cloudera Machine Learning manages everything from data preparation to MLOps, to predictive reporting.CML exposes a REST API that you can use to perform operations related to projects, jobs, and runs. You can use API commands to integrate CML with third-party workflow tools or to control CML from the command line.In this example we will showcase how to use APIv2 to programmatically register an XGBoost experiment via MLFlow Tracking, Registry, and deploy it as a model endpoint in CML.MLOps in CMLCML has extensive MLOps features and a set of model and lifecycle management capabilities to enable the repeatable, transparent, and governed approaches necessary for scaling model deployments and

PurposeDetect updates to S3 files and insert the updated files into Aurora PostgreSQL with NiFiData flowThe finished dataflowProcess1.) Download the dataflow file(JSON file Import_S3_To_Aurora_PostgreSQL.json)2.) Create a new processor group.When creating this processor group, choose the following JSON file to upload.Step 1: Choose processor group Step 2 : Step 3 : Finish upload. 3.) Install JDBC Driverwget https://jdbc.postgresql.org/download/postgresql-42.7.3.jarmkdir /tmp/nifimv postgresql-42.7.3.jar /tmp/nifi/4.) Set parameters in NiFiSet ListS3 parameters  S3 Access Key set    The values input was protected as access key/values are sensitive values.Only "Sensitive value set" be shown. 4.2) Start the controllver service AWSCredentialsProviderControllerService (for saving AWS sensitive values)  4.3) Start the CSVReader Controller service   4.4) Start the JDBC Connection pool（DBCPConnectionPool-postgreSQL）service 5. ) Set bu

When we develop applications for the Cloudera Data Platform, it is quite often necessary to use third-party libraries like NumPy, SciPy, Pandas, etc, or even different versions of existing components, such as Python.  On the other hand, installing and maintaining Python environments can be a complex and time-consuming task for a system administrator and extra caution needs to be taken when we talk about installing other versions of Python and its modules at the operating system level due to the requirements of Cloudera Manager and Cloudera Runtime. You could also choose to install Python virtual environments, but that would still require effort to keep all cluster nodes up to date. Running python in virtual environments in YARN mode requires extra development effort and a significant increase in the total application size due to dependencies. Another excellent option is to distribute Anaconda as a parcel, but be aware that generating custom parcels requires the Anaconda Enterpri

Introduction:In large Hadoop clusters, efficiently managing block replication and decommissioning of DataNodes is crucial for maintaining system performance and reliability. However, updating Namenode configuration parameters to optimize these processes often requires a Namenode restart, causing downtime and potential disruptions to cluster operations. In this article, we'll explore a procedure to expedite block replication and DataNode decommissioning in HDFS without the need for a Namenode restart. Procedure:Identify Namenode Process Directory:Locate the Namenode process directory for the current active Namenode. This directory typically resides in /var/run/cloudera-scm-agent/process/ followed by a folder that looks like "###-hdfs-NAMENODE"Modify Configuration Parameters:Edit the hdfs-site.xml file in the Namenode process directory.Adjust the following parameters to the recommended values:dfs.namenode.replication.max-streams: Increase to a recommended value (e.g., 100).dfs.namen

I've run into this issue a number of time over the years. It's typically associated with environments using Active Directory or FreeIPA for Kerberos authentication. The tell-tale of this problem is this: even though an interactive kinit (using a password) works for a user, she/he cannot authenticate with a keytab, getting the error: "kinit: Preauthentication failed while getting initial credentials". The problem can affect some users but not others when using Active Directory. NOTE: The problem described below is not the root cause of all the "kinit: Preauthentication failed while getting initial credentials" errors. In these cases, the keytab is typically created with the ktutil utility, using the AES encryption type. The following ktutil commands are typically used to generate the keytab: addent -password -p <username> -k 0 -e aes256-ctswrite_kt <username>.keytab  When the keytab is created, a "salt" is used to encrypt the contents of the keytab.

Cloudera documentation is now available at https://docs.cloudera.com/:The CDP documentation is divided in the following sections corresponding to CDP services and components:Management Console Workload ManagerData CatalogReplication ManagerData HubData WarehouseMachine LearningCloudera RuntimeCloudera ManagerEach of these documentation sections includes its own Release Notes document. Furthermore, when using CDP, you can access contextual help by clicking on the Help icon in the bottom left corner:

Purpose: Access CDW Iceberg table with Snowflake Architecture Image source: Configure an external volume for Iceberg tables Initiate Configure an External Volume in Snowflake Step 1: Configure access permissions for the S3 bucket Create a policy in AWS IAM. I created a policy called zzeng-Snowflake-ext:{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:GetObjectVersion", "s3:DeleteObject", "s3:DeleteObjectVersion" ], "Resource": "arn:aws:s3:::<my-bucket>/data/zzeng/*" }, { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetBucketLocation" ], "Resource": "arn:aws:s3:::<my-bucket>", "Condition": { "StringLike": { "s3:prefix": [

Goal : Use CLI to create / import a dataflow in CDF-PC Then, run Teraform to create/import Dataflow into CDF-PC. Process :  Install CDP CLI (I am using MacOS, so I install cdpcli in this way. pip3 install cdpcliThen confirm it's installed correctly[zzeng@zeng-mbp ~]$ ll ~/Library/Python/3.9/bin/ | grep cdp-rwxr-xr-x@ 1 zzeng staff 250 Mar 4 12:22 cdp-rwxr-xr-x@ 1 zzeng staff 250 Mar 4 12:22 cdp_completer[zzeng@zeng-mbp ~]$ export PATH="$HOME/Library/Python/3.9/bin:$PATH"[zzeng@zeng-mbp ~]$ cdp --version0.9.107[zzeng@zeng-mbp ~]$ Configure CLI: (Reference Link) from [Management Console] ->[User] -> [Profile] Then I can find your API key here. If it does not exist, create one. Then, configure the API key in my MacOS (Reference):cdp configure After this config,  check the latest status:[zzeng@zeng-mbp ~]$ cdp iam get-user{ "user": { "userId": ***** "status": "ACTIVE", "workloadPasswordDetails": { "isPasswordSet": t

Cloudera Data Platform provides two complementing services for data processing. Cloudera Data Engineering for preparing and organising data to be consumed by Data Scientists in Cloudera AI. This article provides a high level example of how to call CDE from CML in Python. ## Part 0: Imports# Requests will be used to make REST requests to the CDE service# JSON is used to manage payloads and responses# REST calls are made over HTTP and use BasicAuth# cde_endpoint is set to the high level URI to the CDE cluster endpointimport requestsimport jsonfrom requests.auth import HTTPBasicAuthcde_endpoint='cde-<cluster>.cloudera.site' The first step is establishing the credentials required to call the API. This is done by calling the Knox proxy and requesting a token. The token is then passed using the "Authorization" header. ## Part 1: Connect to Knox service.# Retrieve the JWT token and parse it for the 'access_token' part.# Need to pass workload username and password which is set as a proj

I must be missing something obvious, but how do I change the email address associated with my cloudera.com account? Thank you,David

ObjectiveCDE is the Cloudera Data Engineering Service, a containerized managed service for Cloudera Data Platform designed for Large Scale Batch and Streaming Pipelines with Spark, Airflow and Iceberg. It allows you to submit batch jobs to auto-scaling virtual clusters. As a Cloud-Native service, CDE enables you to spend more time on your applications, and less time on infrastructure.CDE allows you to create, manage, and schedule Apache Spark jobs without the overhead of creating and maintaining Spark clusters. With CDE, you define virtual clusters with a range of CPU and memory resources, and the cluster scales up and down as needed to run your Spark workloads, helping to control your cloud costs.CDE Sessions are short-lived development environment used by Data Engineers and Analysts in order to iterate upon and build Spark workloads, prototype pipelines, or simply explore data. You can use CDE Sessions in CDE Virtual Clusters of type "All Purpose - Tier 2".In this example we will use

Spark Scala Version Compatibility Matrix 1. Introduction Apache Spark being a widely used framework for big data processing, relies heavily on Scala as its primary programming language. Ensuring compatibility between different versions of Spark and Scala is essential for developers to leverage the latest features and optimizations while maintaining stability in their Spark applications.  In this article, we'll provide a comprehensive overview of the compatibility matrix between different versions of Spark and Scala, helping developers choose the right combination for their projects. Key Considerations: Several factors influence compatibility between Spark and Scala versions: API changes: New features or modifications in Spark APIs might require specific Scala versions for proper compilation and execution. Library dependencies: Third-party libraries used within your Spark application might have compatibility requirements with both Spark and Scala versions. Community support:

Spark and Java versions Supportability Matrix 1. Introduction: Apache Spark is a powerful open-source distributed computing system widely used for big data processing and analytics. However, choosing the right Java version for your Spark application is crucial for optimal performance, security, and compatibility.  This article dives deep into the officially supported Java versions for Spark, along with helpful advice on choosing the right one for your project. 2. Matrix Table Spark Version Supported Java Version(s) Java 8 Java 11 Java 17 Java 21 Deprecated Java Version(s) 3.5.1 Java 8*/11/17 Yes Yes Yes No Java 8 prior to version 8u371 support is deprecated 3.5.0 Java 8*/11/17 Yes Yes Yes No Java 8 prior to version 8u371 support is deprecated 3.4.2 Java 8*/11/17 Yes Yes Yes No Java 8 prior to version 8u362 support is deprecated 3.4.1 Java 8*/11/17 Yes Yes Yes No Java 8 prior to version 8u362 support is

What is DataFlint DataFlint is an open-source D-APM (Data-Application Performance Monitoring) for Apache Spark, built for big data engineers. For more information see dataflint.gitbook.io/dataflint-for-spark or the git-repo itself  How to Integrate DataFlint in CDP DataFlint supports spark 3.2+, so you will need a spark3 parcel  For spark-submit see DataFlint:    spark-submit--packages io.dataflint:spark_2.12:0.3.1 \--conf spark.plugins=io.dataflint.spark.SparkDataflintPlugin \...   To Install on Spark History Server: download DataFlint jar for scala2.12  Copy it to all servers that have an instance of spark-history Role to spark 3 parcel dir:/opt/cloudera/parcels/SPARK3/lib/spark3/jars​Beware /opt/cloudera/parcels/SPARK3 is a symlink to the current SPARK3 active parcel and is controlled by Cloudera Manager. Go to CM >cluster >Spark 3 >Configuration >spark3-conf/spark-history-server.conf_role_safety_valv

Hi, Am unable to change my community nickname, is there any way to change it? Thanks

This article describes various "recipes" on how to accomplish certain tasks with the NiFi processor ExecuteScript, with examples given in Groovy, Jython, Javascript (Nashorn), and JRuby. Recipes in this article series include: Part 1 - Introduction to the NiFi API and FlowFiles Getting a flow file from an incoming queue Creating new flow files Working with flow file attributes Transferring a flow file Logging Part 2 - FlowFile I/O and Error Handling Reading from a flow file Writing to a flow file Reading and writing to/from a flow file Error Handling Part 3 - Advanced Features Using Dynamic Properties Adding Modules State Management Accessing Controller Services Introduction ExecuteScript is a versatile processor that allows the user to code custom logic in a programming language that will be executed each time the ExecuteScript processor is triggered. The following variable bindings are provided to the script to enable access to NiFi components: session: This is a reference to

SummaryIn this article, we learn how to use CML to automate Machine Learning workflows between development and production environments using an automation toolchain such as Gitlab. This solution can be extended to other platforms such as Github( using Github actions) or Atlassian (using Bitbucket pipelines).Some prior understanding of how the DevOps pipeline works and how it can be applied in Machine learning is assumed here. There are some good resources online that can be referred for a deep-dive, but just enough information is provided below. A (very) brief Introduction to CI-CD for Machine LearningA DevOps pipeline is a set of automated processes and tools that allows software development teams to build and deploy code to a production environment. A few of the important components of the DevOps pipeline include Continuous Integration (CI) and Continuous Delivery and Deployment (CD), where:  Continuous Integration is the practice or ability to make frequent commi