Community Articles

Issue During the installation of the latest version of mlflow for the custom runtime engine in CDSW 1.10.x, it's still using cmladdon’s version rather than the version installed via pip. For example, in one case, the error recorded was the following: ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflict mlflow-cml-plugin 0.0.1 requires protobuf==3.19,4, but you have protobuf 4.24.0 which is incompatible. 
onnx 1.12.0 requires protobuf<=3.20.1,>=3.12.2, but you have protobuf 4.24.0 which is incompatible.ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflict mlflow-cml-plugin 0.0.1 requires protobuf==3.19,4, but you have protobuf 4.24.0 which is incompatible. 
onnx 1.12.0 requires protobuf<=3.20.1,>=3.12.2, but you have protobuf 4.24.0 which

Hi Everyone, I ran into a situation where I have to the following XML Input:     <workspace id="1234"><documents totalCount="2"><document id="5555" title="document1" revision="1"><file name="file1.pdf" type="pdf"/><attributes> <attribute name="custAtt1" value="v1"/> <attribute name="custAtt2" value="v2"/> <attribute name="custAtt3" value=""/></attributes></document><document id="6666" title="document2" revision="0"> <file name="file2.xlsx" type="xlsx"/> <attributes> <attribute name="custAtt1" value="v1"/> <attribute name="custAtt2" value="v2"/> <attribute name="custAtt3" value="v3"/> </attributes></document></documents></workspace>     Here, each /workspace/documents/document record needed to be split and transformed into JSON, where: Each document object needs to have the workspace ID it belongs to. Each document attribute value will be as

Introduction: This article is designed to get the reader to call a deployed CML model from Apache NiFi or Cloudera Data Flow as quickly as possible.  It will require using one of Cloudera's Applied Machine Learning Prototypes (AMPs) on Customer Churn and the Apache NiFi Flow provided in this article.  (Git Repo Here) Prerequisites: Apache NiFi instance is running. CML's AMP Customer Churn App is running with a Churn Model deployed.  Download the user data file and Apache Nifi flow from this repo.  The user data file needs to be added to the file location where NiFi's GetFile Processor has permissions.  Launch CML Churn AMP In the AMP catalog, select Churn Modeling with skit-learn, and configure a new project.  This will take 5-7 minutes to complete.  You will need to deploy a model inside of CML, which isn't in the scope of this article.  Once you finish, you will be able to see the following screen, which will provide the URL location an

In this article, we will learn how to run scheduled ETL workloads on CDE ( Cloudera Data Engineering )  and CDW ( Cloudera Data Warehouse) using Airflow.   CDE 1.4 release provides the capability to schedule Airflow jobs (shipped along with CDE). If we need to learn more about Airflow, refer to the Airflow website.  This release adds two Airflow operators: CDEJobRunOperator and CDWOperator. The CDEJobRunOperator behind the scene calls the jobs API of CDE to run Spark jobs. More details on jobs API can be found here.   The CDWOperator behind the scene makes use of JDBC connection to interact with CDW Hive Virtual warehouse or Impala Virtual warehouse.   In this article to demonstrate ETL workload, we will be reading CSV from AWS S3 bucket using Spark, perform a transformation (via CDEJobRunOperator) and load the processed data to CDW warehouse using Airflow CDWOperator. Pre-Requisite  CDP environment is created and CDW, CDE experience are provisio

This is a simple command that can be used to run via a browser to change the log level without needing a restart.For Hbase Master:http://<SERVER-HOSTNAME>:16010/logLevel?log=org.apache.hadoop.hbase& level=<DESIRED-LOG-LEVEL>For HBase Region Server:http://<SERVER-HOSTNAME>:16030/logLevel?log=org.apache.hadoop.hbase& level=<DESIRED-LOG-LEVEL> **NOTE:Please make sure there is no space between '&' and 'level'. There seems to be a formatting issue in HCC because of which a space was introduced in the syntax.<SERVER-HOSTNAME> : Represents the hostname of the server where the daemon is running.<DESIRED-LOG-LEVEL>: Represent the log level that is desired by the user to have. Ex. INFO, DEBUG,..

Spark3 HBase Integration This blog post will guide you through the process of integrating Spark 3 with HBase, providing you with valuable insights and step-by-step instructions.Note: This blog post will cover both Spark3 and HBase services only when they exists in the same cluster. If HBase is in remote  cluster, you can go through the hbase-configure-spark-connector-remote-cluster tutorial.PrerequisitesTo use HBase-Spark3 connector, we need to ensure that CDP version is 7.1.7 SP1 Spark3 parcel or above.Ensure that every Spark node has the HBase Master, Region Server, or Gateway role assigned to it. If no HBase role is assigned to a Spark node, add the HBase Gateway role to it, which ensures that the HBase configuration files are available on the Spark node.HBase Integration with Spark3HBase integration with Spark can be achieved using the HBase-Spark Connector, which provides a seamless way to interact with HBase from within Spark applicati

Description Repo Name: cml_vscode_with_python310Repo Info: VsCode using Python 3.10, instead of the existing VsCode example using Python 3.7. Repo URL: https://github.com/ryancicak/cml_vscode_with_python310 

I recently received a question "We'd like to have Python and R installed in the docker image for a package.  Is it possible to have such customized docker with python >=3.7, R>=4.0 and CUDA==11.7?" I'd like to go through how I accomplished this using the pbj-workbench-r4.1-standard.Dockerfile.  This runtime has the essentials (Python and R) except for CUDA (NVIDIA GPU).  Therefore, we'll use pbj-workbench-r4.1-standard.Dockerfile as a base image.  It's important to note that CML requires a runtime kernel that needs to be either R or Python, it can't be both.  I'll go through three different iterations of this image: Iteration 1 - Since this image already has R and Python installed, we'll change the runtime NOT to use Rryancicak/pbj_r-and-python:v9 -> Python 3.10 (GitHub Tag: python/3.10/standard_dockerhub_v9)https://github.com/ryancicak/pbj_r-and-python/blob/python/3.10/standard_dockerhub_v9/Dockerfile Iteration 2 - G

Spark Streaming Graceful shutdownSpark Streaming jobs are long-running jobs and their tasks need to be executed 7*24 hours, but there are cases like upgrading the code or updating the Spark configuration, where the program needs to be actively stopped. However, for distributed programs, there is no way to kill processes one by one. It is very important to shut down all configurations gracefully.There are two ways to shut down streaming applications:Non-graceful shutdownGraceful shutdown1. Non-graceful shutdownA non-graceful shutdown is a violent shutdown process. Stopping Spark streaming violently may cause problems. For example, if your data source is Kafka, and a batch of data has been loaded into Spark streaming for processing. If it is forced to stop, consumption will be repeated or some data will be lost at the next startup. Disadvantages:Spark Streaming works based on the micro-batch mechanism, RDD is generated according to the interval time. If a violent shut

OBJECTIVE:Provide a quick-start guide for using the Jolt language within a NiFi JoltTransform (JoltTransformJSON or JoltTransformRecord).OVERVIEW:The NiFi JoltTransform uses the powerful Jolt language to parse JSON. Combined with the NiFi Schema Registry, this gives NiFi the ability to traverse, recurse, transform, and modify nearly any data format that can be described in AVRO or, using JSON as an intermediary step.Although the language itself is open-source, and some documentation is available in the JavaDoc, this article can provide a starting point for understanding basic Jolt operations.PREREQUISITES:HDF 3.0 or later (NiFi 1.2.0.3 or later)BASICS OF JOLT:Simplified OverviewThe JoltTransform applies a set of transformations described in a JSON specification to an input JSON document and generates a new output JSON document.Jolt SpecificationOverviewA Jolt Specification is a JSON structure that contains two root elements:operation (string): shift, sort, cardinality, modify-default-b

Starting from NiFi 1.14, you can specify the TLS ciphers to be used by NiFi web service by using below property: nifi.web.https.ciphersuites.include You can also specify the TLS Ciphers to be excluded by using below property: nifi.web.https.ciphersuites.exclude This enhancement is part of Apache Jira NIFI-3580.    For Cloudera CFM, these properties are exposed (and usable) from CFM 2.1.3 and above.   Note: The values for above properties should take the form of a comma-separated list of common cipher names as specified here. The regular expressions (for example ^.*GCM_SHA256$) may also be specified.

Spark3 Kudu Integration   This blog post will guide you through the process of integrating Spark 3 with Kudu, providing you with valuable insights and step-by-step instructions. Apache Kudu Kudu is a distributed columnar storage engine optimized for OLAP workloads. Kudu runs on commodity hardware, is horizontally scalable, and supports highly available operations. Kudu Integration with Spark Apache Kudu can be integrated with Apache Spark using the built-in Spark-SQL-Kudu library. This integration allows us to read from and write to Kudu tables directly from Spark applications using Spark SQL. Kudu integrates with Spark through the Data Source API as of version 1.0.0. To integrate Kudu with Spark3, we need to use kudu-spark3_2.12 library.  Below are the step-by-step instructions to seamlessly integrate Kudu with Spark3:   Impala/Kudu: Step 1: Launch the impala-shell Go to the Cloudera Manager > Impala > Status > Copy the Impala Shell Com

When starting NiFi, it uses the default staging path which is /home/NiFi/. This path needs the Execute permissions for NiFi binaries to avoid NiFi from failing to starting . The below error would also indicate the cause of this issue;   Caused by: java.lang.UnsatisfiedLinkError: /home/nifi/.cache/JNA/temp/jna3526468256198020468.tmp: /home/nifi/.cache/JNA/temp/jna3526468256198020468.tmp: failed to map segment from shared object: Operation not permitted Solution In case of security hardening (noexec) for /home, you need to change the default staging directory in NiFi by using the below parameters: CM > NIFI > ConfigsNiFi Node Advanced Configuration Snippet (Safety Valve) for staging/bootstrap.conf.xmladd the following:Name: java.arg.jnaValue: -Djna.tmpdir=/NEWPATHRestart NIFI services.

Windows Event Logs are stored and exported in the evtx format. This makes them difficult to work with in a programmatic fashion, especially outside the Windows ecosystem. In order to facilitate the processing of Windows event data in Apache NiFi, a processor capable of parsing evtx files has been created.It works using a parser ported from Python into pure Java and modified to read the file one chunk (64k) at a time to allow for processing large files without pulling the entire input FlowFile into memory.The Processor expects incoming FlowFiles with evtx files as their contents. It only requires the user to specify a desired granularity of output. This allows the user to decide how the evtx file is broken up into individual FlowFiles with the event records translated to XML.ParseEvtx has 4 output relationships. It will output the unmodified FlowFile to the original relationship. Any malformed chunks encountered will be sent to the bad chunk relationship. Regardless of the configuration

IntroductionRecently I was asked how to monitor and alert flowfile count in a connection queue when it exceeds a predefined threshold, I am trying to capture the steps to implement the same.Prerequisites1) To test this, Make sure HDF-2.x version of NiFi is up an running.2) You Already have a connection with data queued in it(say more than 20 flowfiles). Else you can create one like below:3) Make a note of the Connection name and uuid to be monitored:Creating a Flow to Monitor Connection Queue Count.1) Drop a GenerateFlowFile processor to the canvas and make "Run Schedule" 60 sec so we dont execute the flow to often.2) Drop an UpdateAttribute processor, connect GenerateFlowFile's success relation to it and add below properties to it( the connection uuid noted above, threshold say 20, your NiFi host and port):CONNECTION_UUID : dcbee9dd-0159-1000-45a7-8306c28f2786COUNT : 20NIFI_HOST : localhostNIFI_PORT : 80803) Drop a InvokeHTTP processor to the canvas, connect Upda

After upgrading to CFM 2.1.5 SP1from an older version, PublishKafkaRecord_1_0 stops working with the below error:   PublishKafka_1_0[id=36f1cffe-503d-3181-a562-ae216b5616b5] Processing halted: yielding [1 sec] java.lang.IllegalArgumentException: SaslMechanism value [null] not found at org.apache.nifi.kafka.shared.property.SaslMechanism.lambda$getSaslMechanism$1(SaslMechanism.java:57) at java.base/java.util.Optional.orElseThrow(Optional.java:408) at org.apache.nifi.kafka.shared.property.SaslMechanism.getSaslMechanism(SaslMechanism.java:57) at org.apache.nifi.kafka.shared.login.DelegatingLoginConfigProvider.getConfiguration(DelegatingLoginConfigProvider.java:51) at org.apache.nifi.kafka.shared.property.provider.StandardKafkaPropertyProvider.setSecurityProperties(StandardKafkaPropertyProvider.java:86) at org.apache.nifi.kafka.shared.property.provider.StandardKafkaPropertyProvider.getProperties(StandardKafkaPropertyProvider.java:75) This happens due an improvement performed in N

Summary As Kafka works with many log segment files and network connections, the Maximum Process File Descriptors setting may need to be increased in some cases in production deployments, if a broker hosts many partitions Warnings observed Concerning : Open file descriptors: 18,646. File descriptor limit: 32,768. Percentage in use: 56.90%. Warning threshold: 50.00%. In case of such open file descriptor warnings in Kafka for CDP, you need to increase file descriptors in Kafka cluster. As per the guide [1], below formula is used (number of partitions)*(partition size / segment size)Details about these parameters are as follows:a. Number of partitions > it can be seen if you go to SMM and brokers tab. Alternatively, you can simply perform a file count of the kafka data directory. You can find this directory in CM > Kafka > log.dirsb. Partition size > it is the size of log.dirs directory, you can check the disk usage of log.dirs using du -sh command to find this size.c. Segment

High level Problem Explanation The occurrence of the "block missing" and "block corruption" alerts is observed in the Cloudera Manager dashboard or the namenode webui. While there isn't a common cause, these alerts can be due to any of the outlined issues.   Detailed Technical Explanation When the namenode does not receive reports for all of the block replicas of a file, it marks those blocks as missing. Similarly, when all the block replicas are corrupted, the namenode marks them as corrupt. However, if at least one of the block replica(1/3) is reported as being healthy, then there is no missing or corruption alert. There are multiple scenarios in which blocks can be marked as missing and corrupt.   Scenario 1 During a major upgrade, users may encounter block missing errors. This issue arises due to insufficient datanode heap.   The upgrade process involves migrating the disk layout and requires additional datanode heap space. users who are upgrading to releases C

Objective This tutorial walks you through how to secure a NiFi instance using client certificates, configure access policies in NiFi, and then how to integrate it with a secured NiFi Registry to utilize versioned flows.A video version of this tutorial can be seen here: https://youtu.be/DSO12fhnZ90 Note: This tutorial assumes you have completed Setting Up a Secure Apache NiFi Registry which resulted in a NiFi Registry instance that has been secured with client certificates for two users: "sys_admin" and "test_user".Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.5.0 Apache NiFi Registry 0.1.0 Apache NiFi Toolkit 1.5.0Secure NiFi ConfigurationDownload & Extract NiFi Tarball Download the tarball for the 1.5.0 Apache NiFi release: nifi-1.5.0-bin.tar.gz Extract the tar: tar xzvf nifi-1.5.0-bin.tar.gzNiFi Configuration From the article Setting Up a Secure Apache NiFi Registry, the TLS Toolkit was used to

What is Apache Iceberg?Apache Iceberg is an Open table format developed by the open source community for high performance analytics on petabyte scale data sets. Apache Iceberg is engine agnostic and also supports SQL commands, that is Hive, Spark, Impala, and so on can all be used to work with Iceberg tables. It is quickly becoming the format of choice for large data sets of sizes anywhere from 100s of TB to PBs across the data industry. What are the benefits of Apache Iceberg?By maintaining table metadata and state in manifest files and snapshots on the object store, Iceberg avoids expensive calls to a separate metastore like HMS to retrieve partition information which results in 10x faster query planning. Its main features include Hidden partitioning, In-place partition evolution, Time travel, Out-of-the box Data compaction, and Update, Delete, Merge operations in v2. What is Apache Ozone?Apache Ozone is a highly scalable, highly available, and secure object store that can

The Cloudera is a hybrid data platform designed to deliver faster and easier data management, analytics and AI at enterprise scale. Cloudera AI is one of CDP’s Cloud Native Data Services designed to enable secure, governed Data Science.    With immediate access to enterprise data pipelines, scalable compute resources, and access to preferred tools, Data Scientists and Engineers use CML to streamline the process of getting analytic workloads into production and intelligently manage machine learning use cases and MLOps processes.   While CML Data Scientists spend most of their time prototyping and productionizing models in CML Projects, the CML Admin needs to be familiar with the CML Workspace, its basic architecture and how it allocates Cloud resources.    With this article we will share some foundational concepts in order to help CML Administrators better understand how to size Workspaces.   What is a Workspace? CML workloads are executed within Workspace

In the last few years, Spark has become one of the default choices for Data Analysis, Engineering, and Machine Learning in Cloudera AI. CML is the Cloud-Native autoscaling service in the Cloudera Data Platform (CDP) for secure and governed Machine Learning and interactive Data Analysis at Enterprise Scale.In this article, we will share important considerations when using Spark in CML in order to help Developers and Data Scientists in their daily usage.  CML SessionsA CML Workspace provides an autoscaling Kubernetes cluster for running workloads such as interactive notebook Sessions, as well as Batch Jobs, Experiments, Applications, and Models. These are available in the CML Project, a construct to isolate users in their own environments.  Workspace implementation details are outside of this article’s scope. Luckily neither the CML User nor Admin is required to know these or operate Kubernetes at all. Under the covers, CDP provides the necessary infrastructure and in

Have you stumbled upon existing articles on only reading from a managed Hive table using R within CML, but you really want to read/write to an external table (not a Hive Managed table)?  Then you've come to the right place!     There are complexities to reading/writing to a Hive-managed table since ACID is involved, as the HWC piece solves.  In 2023, if you're looking for ACID capabilities then using Iceberg table format instead of Hive-managed tables is recommended anyways. The Iceberg table format is fully supported throughout CDP.     Let's get on with the steps to easily read/write using R to an external table within CML!  Again, without the need for HWC.     Step 1: Created two tables (external) and inserted records *In this case, I used Cloudera Data Warehouse with an Impala Virtual Warehouse   create database cloudera_r;create external table cloudera_r.people_table (id int, name string) location 's3a://ryanc

Today I finally found the time and subject matter to start an article showcasing how I am working to Operationalize NiFI Flows with Cloudera's CDP Public Cloud DataFlow Data Service.  I have been so busy working with my Cloudera accounts while training up on all things CDP I just have not had a chance to write a community article.  I am very excited to share my experience and this incredible modernization Cloudera has provided on top of NiFi.   Traditionally, large scale NiFi Environments and the Nifi Canvas contain all the NiFi process groups and data flows.  As you can see here, this gets visually complicated:      Operating NiFi in this manner makes operations around the data flows on this NiFI canvas technically complicated.   Even with user access/auth via Ranger, the root level canvas would still show the entire cluster's Process Groups.  If you are an Operations owner of NiFi with hundreds of flows you did not make, it wou

Overview      This article describes a NiFi flow that consumes data from Tanium API endpoints. This article can be useful for those who want to consume Tanium APIs but is also useful for anyone wanting to pull from an API that may not instantly return a full set of results, and then when the data is ready, to pull the data using pagination. Flow Beginning      The flow shows two data subject areas. The data subjects are determined by the Tanium question and it is assumed that saved questions have been defined within the Tanium source. Since the data results are combined into a funnel and all processed in the same way (due to the fact that the Tanium API always returns results with the same structure), additional data subjects could be added in the same way and joined into the funnel.      Each data subject is pulled from three different Tanium endpoints (DEV, TEST, PROD environments). This is to simulate having three domains