Resources

Spark3 Kudu Integration   This blog post will guide you through the process of integrating Spark 3 with Kudu, providing you with valuable insights and step-by-step instructions. Apache Kudu Kudu is a distributed columnar storage engine optimized for OLAP workloads. Kudu runs on commodity hardware, is horizontally scalable, and supports highly available operations. Kudu Integration with Spark Apache Kudu can be integrated with Apache Spark using the built-in Spark-SQL-Kudu library. This integration allows us to read from and write to Kudu tables directly from Spark applications using Spark SQL. Kudu integrates with Spark through the Data Source API as of version 1.0.0. To integrate Kudu with Spark3, we need to use kudu-spark3_2.12 library.  Below are the step-by-step instructions to seamlessly integrate Kudu with Spark3:   Impala/Kudu: Step 1: Launch the impala-shell Go to the Cloudera Manager > Impala > Status > Copy the Impala Shell Com

When starting NiFi, it uses the default staging path which is /home/NiFi/. This path needs the Execute permissions for NiFi binaries to avoid NiFi from failing to starting . The below error would also indicate the cause of this issue;   Caused by: java.lang.UnsatisfiedLinkError: /home/nifi/.cache/JNA/temp/jna3526468256198020468.tmp: /home/nifi/.cache/JNA/temp/jna3526468256198020468.tmp: failed to map segment from shared object: Operation not permitted Solution In case of security hardening (noexec) for /home, you need to change the default staging directory in NiFi by using the below parameters: CM > NIFI > ConfigsNiFi Node Advanced Configuration Snippet (Safety Valve) for staging/bootstrap.conf.xmladd the following:Name: java.arg.jnaValue: -Djna.tmpdir=/NEWPATHRestart NIFI services.

Windows Event Logs are stored and exported in the evtx format. This makes them difficult to work with in a programmatic fashion, especially outside the Windows ecosystem. In order to facilitate the processing of Windows event data in Apache NiFi, a processor capable of parsing evtx files has been created.It works using a parser ported from Python into pure Java and modified to read the file one chunk (64k) at a time to allow for processing large files without pulling the entire input FlowFile into memory.The Processor expects incoming FlowFiles with evtx files as their contents. It only requires the user to specify a desired granularity of output. This allows the user to decide how the evtx file is broken up into individual FlowFiles with the event records translated to XML.ParseEvtx has 4 output relationships. It will output the unmodified FlowFile to the original relationship. Any malformed chunks encountered will be sent to the bad chunk relationship. Regardless of the configuration

IntroductionRecently I was asked how to monitor and alert flowfile count in a connection queue when it exceeds a predefined threshold, I am trying to capture the steps to implement the same.Prerequisites1) To test this, Make sure HDF-2.x version of NiFi is up an running.2) You Already have a connection with data queued in it(say more than 20 flowfiles). Else you can create one like below:3) Make a note of the Connection name and uuid to be monitored:Creating a Flow to Monitor Connection Queue Count.1) Drop a GenerateFlowFile processor to the canvas and make "Run Schedule" 60 sec so we dont execute the flow to often.2) Drop an UpdateAttribute processor, connect GenerateFlowFile's success relation to it and add below properties to it( the connection uuid noted above, threshold say 20, your NiFi host and port):CONNECTION_UUID : dcbee9dd-0159-1000-45a7-8306c28f2786COUNT : 20NIFI_HOST : localhostNIFI_PORT : 80803) Drop a InvokeHTTP processor to the canvas, connect Upda

After upgrading to CFM 2.1.5 SP1from an older version, PublishKafkaRecord_1_0 stops working with the below error:   PublishKafka_1_0[id=36f1cffe-503d-3181-a562-ae216b5616b5] Processing halted: yielding [1 sec] java.lang.IllegalArgumentException: SaslMechanism value [null] not found at org.apache.nifi.kafka.shared.property.SaslMechanism.lambda$getSaslMechanism$1(SaslMechanism.java:57) at java.base/java.util.Optional.orElseThrow(Optional.java:408) at org.apache.nifi.kafka.shared.property.SaslMechanism.getSaslMechanism(SaslMechanism.java:57) at org.apache.nifi.kafka.shared.login.DelegatingLoginConfigProvider.getConfiguration(DelegatingLoginConfigProvider.java:51) at org.apache.nifi.kafka.shared.property.provider.StandardKafkaPropertyProvider.setSecurityProperties(StandardKafkaPropertyProvider.java:86) at org.apache.nifi.kafka.shared.property.provider.StandardKafkaPropertyProvider.getProperties(StandardKafkaPropertyProvider.java:75) This happens due an improvement performed in N

Summary As Kafka works with many log segment files and network connections, the Maximum Process File Descriptors setting may need to be increased in some cases in production deployments, if a broker hosts many partitions Warnings observed Concerning : Open file descriptors: 18,646. File descriptor limit: 32,768. Percentage in use: 56.90%. Warning threshold: 50.00%. In case of such open file descriptor warnings in Kafka for CDP, you need to increase file descriptors in Kafka cluster. As per the guide [1], below formula is used (number of partitions)*(partition size / segment size)Details about these parameters are as follows:a. Number of partitions > it can be seen if you go to SMM and brokers tab. Alternatively, you can simply perform a file count of the kafka data directory. You can find this directory in CM > Kafka > log.dirsb. Partition size > it is the size of log.dirs directory, you can check the disk usage of log.dirs using du -sh command to find this size.c. Segment

High level Problem Explanation The occurrence of the "block missing" and "block corruption" alerts is observed in the Cloudera Manager dashboard or the namenode webui. While there isn't a common cause, these alerts can be due to any of the outlined issues.   Detailed Technical Explanation When the namenode does not receive reports for all of the block replicas of a file, it marks those blocks as missing. Similarly, when all the block replicas are corrupted, the namenode marks them as corrupt. However, if at least one of the block replica(1/3) is reported as being healthy, then there is no missing or corruption alert. There are multiple scenarios in which blocks can be marked as missing and corrupt.   Scenario 1 During a major upgrade, users may encounter block missing errors. This issue arises due to insufficient datanode heap.   The upgrade process involves migrating the disk layout and requires additional datanode heap space. users who are upgrading to releases C

Objective This tutorial walks you through how to secure a NiFi instance using client certificates, configure access policies in NiFi, and then how to integrate it with a secured NiFi Registry to utilize versioned flows.A video version of this tutorial can be seen here: https://youtu.be/DSO12fhnZ90 Note: This tutorial assumes you have completed Setting Up a Secure Apache NiFi Registry which resulted in a NiFi Registry instance that has been secured with client certificates for two users: "sys_admin" and "test_user".Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.5.0 Apache NiFi Registry 0.1.0 Apache NiFi Toolkit 1.5.0Secure NiFi ConfigurationDownload & Extract NiFi Tarball Download the tarball for the 1.5.0 Apache NiFi release: nifi-1.5.0-bin.tar.gz Extract the tar: tar xzvf nifi-1.5.0-bin.tar.gzNiFi Configuration From the article Setting Up a Secure Apache NiFi Registry, the TLS Toolkit was used to

What is Apache Iceberg?Apache Iceberg is an Open table format developed by the open source community for high performance analytics on petabyte scale data sets. Apache Iceberg is engine agnostic and also supports SQL commands, that is Hive, Spark, Impala, and so on can all be used to work with Iceberg tables. It is quickly becoming the format of choice for large data sets of sizes anywhere from 100s of TB to PBs across the data industry. What are the benefits of Apache Iceberg?By maintaining table metadata and state in manifest files and snapshots on the object store, Iceberg avoids expensive calls to a separate metastore like HMS to retrieve partition information which results in 10x faster query planning. Its main features include Hidden partitioning, In-place partition evolution, Time travel, Out-of-the box Data compaction, and Update, Delete, Merge operations in v2. What is Apache Ozone?Apache Ozone is a highly scalable, highly available, and secure object store that can

The Cloudera is a hybrid data platform designed to deliver faster and easier data management, analytics and AI at enterprise scale. Cloudera AI is one of CDP’s Cloud Native Data Services designed to enable secure, governed Data Science.    With immediate access to enterprise data pipelines, scalable compute resources, and access to preferred tools, Data Scientists and Engineers use CML to streamline the process of getting analytic workloads into production and intelligently manage machine learning use cases and MLOps processes.   While CML Data Scientists spend most of their time prototyping and productionizing models in CML Projects, the CML Admin needs to be familiar with the CML Workspace, its basic architecture and how it allocates Cloud resources.    With this article we will share some foundational concepts in order to help CML Administrators better understand how to size Workspaces.   What is a Workspace? CML workloads are executed within Workspace

In the last few years, Spark has become one of the default choices for Data Analysis, Engineering, and Machine Learning in Cloudera AI. CML is the Cloud-Native autoscaling service in the Cloudera Data Platform (CDP) for secure and governed Machine Learning and interactive Data Analysis at Enterprise Scale.In this article, we will share important considerations when using Spark in CML in order to help Developers and Data Scientists in their daily usage.  CML SessionsA CML Workspace provides an autoscaling Kubernetes cluster for running workloads such as interactive notebook Sessions, as well as Batch Jobs, Experiments, Applications, and Models. These are available in the CML Project, a construct to isolate users in their own environments.  Workspace implementation details are outside of this article’s scope. Luckily neither the CML User nor Admin is required to know these or operate Kubernetes at all. Under the covers, CDP provides the necessary infrastructure and in

Have you stumbled upon existing articles on only reading from a managed Hive table using R within CML, but you really want to read/write to an external table (not a Hive Managed table)?  Then you've come to the right place!     There are complexities to reading/writing to a Hive-managed table since ACID is involved, as the HWC piece solves.  In 2023, if you're looking for ACID capabilities then using Iceberg table format instead of Hive-managed tables is recommended anyways. The Iceberg table format is fully supported throughout CDP.     Let's get on with the steps to easily read/write using R to an external table within CML!  Again, without the need for HWC.     Step 1: Created two tables (external) and inserted records *In this case, I used Cloudera Data Warehouse with an Impala Virtual Warehouse   create database cloudera_r;create external table cloudera_r.people_table (id int, name string) location 's3a://ryanc

Today I finally found the time and subject matter to start an article showcasing how I am working to Operationalize NiFI Flows with Cloudera's CDP Public Cloud DataFlow Data Service.  I have been so busy working with my Cloudera accounts while training up on all things CDP I just have not had a chance to write a community article.  I am very excited to share my experience and this incredible modernization Cloudera has provided on top of NiFi.   Traditionally, large scale NiFi Environments and the Nifi Canvas contain all the NiFi process groups and data flows.  As you can see here, this gets visually complicated:      Operating NiFi in this manner makes operations around the data flows on this NiFI canvas technically complicated.   Even with user access/auth via Ranger, the root level canvas would still show the entire cluster's Process Groups.  If you are an Operations owner of NiFi with hundreds of flows you did not make, it wou

Overview      This article describes a NiFi flow that consumes data from Tanium API endpoints. This article can be useful for those who want to consume Tanium APIs but is also useful for anyone wanting to pull from an API that may not instantly return a full set of results, and then when the data is ready, to pull the data using pagination. Flow Beginning      The flow shows two data subject areas. The data subjects are determined by the Tanium question and it is assumed that saved questions have been defined within the Tanium source. Since the data results are combined into a funnel and all processed in the same way (due to the fact that the Tanium API always returns results with the same structure), additional data subjects could be added in the same way and joined into the funnel.      Each data subject is pulled from three different Tanium endpoints (DEV, TEST, PROD environments). This is to simulate having three domains

NiFi - Oracle Connect Processor: ExecuteSQL 1) Prepare Oracle Data (Optional) Prepare the data, and create the Oracle table:   CREATE TABLE USER_MASTER( USER_ID VARCHAR2(8) NOT NULL, DEPT_NO VARCHAR2(8), USER_NAME VARCHAR2(32), CREATED_ON DATE DEFAULT SYSDATE, MODIFIED_ON DATE, CONSTRAINT pk_USER_MASTER PRIMARY KEY(USER_ID));   Insert the data into Oracle Table:   INSERT INTO USER_MASTER (USER_NAME, DEPT_NO, USER_ID) VALUES ('Hayashi　1 Ro','1001','1');INSERT INTO USER_MASTER (USER_NAME, DEPT_NO, USER_ID) VALUES ('Hayashi　2 Ro','1001','2');INSERT INTO USER_MASTER (USER_NAME, DEPT_NO, USER_ID) VALUES ('Hayashi　3 Ro','1001','3');INSERT INTO USER_MASTER (USER_NAME, DEPT_NO, USER_ID) VALUES ('Hayashi　4 Ro','1001','4');INSERT INTO USER_MASTER (USER_NAME, DEPT_NO, USER_ID) VALUES ('Hayashi　5 Ro','1001','5');INSERT INTO USER_MASTER (USER_NAME, DEPT_NO, USER_ID) VALUES ('Hayashi　6 Ro','1001','6');INSERT INTO USER_MASTER (USER_NAME

In a recent release, CDW is now fully integrated with RAZ.  This means once you provision an environment in CDP with RAZ enabled, adding an S3 bucket into CDW follows the same principles as the services within CDP (CML, CDE, COD, CDF, and DataHub).   I was able to add a new S3 bucket called "ryancicak" in two steps, after receiving the 403 Forbidden error within Hue: Two steps:   Step 1 - Since I used the -datalake-admin-role for RAZ, I modified the "datalake-admin-s3-policy to include my new bucket "ryancicak" with the two lines: "arn:aws:s3:::ryancicak","arn:aws:s3:::ryancicak/*"   *Notice I also removed the subdirectory from my cicak-cdp-stack to include the root directory.  This is important to make this modification as well.  Instead of "cicak-cdp-stack/my-data", I state "cicak-cdp-stack" Step 2 - I added a new policy in Ranger under cm_s3 for my bucket "ryancicak", for the users rcicak (me), and Hive.     DONE - When re-r

The Phoenix documentation here leaves out a few pieces in order to make a successful connection to HBase, through the Phoenix Driver. They assume that the connection is from the 'localhost'. May work great, but that's unlikely in the real world.Required Jarsphoenix-client.jar hbase-client.jar (not mentioned in the Phoenix Docs)URL DetailsThe full adbc connection URL syntax for phoenix is:Basic Connections jdbc:phoneix[:zk_quorum][:zk_port][:zk_hbase_path]The "zk_quorum" is a comma separated list of the ZooKeeper Servers.The "zk_port" is the ZooKeeper port.The "zk_hbase_path" is the path used by Hbase to stop information about the instance. On a 'non' kerberized cluster the default zk_hbase_path for HDP is '/hbase-unsecure'.Sample JDBC URLjdbc:phoenix:m1.hdp.local,m2.hdp.local,d1.hdp.local:2181:/hbase-unsecure For Kerberos Cluster Connections jdbc:phoneix[:zk_quorum][:zk_port][:zk_hbase_path][:headless_keytab_file:principal] Sample JDBC URL jdbc:phoenix:m1.hdp.local,m2.hdp.local,d1.h

Cloudera DataFlow Connectors We would love to have you and your fellow colleagues attend this session. Please feel free to share this internally. We look forward to having you join us!Reading from Iceberghttps://github.com/tspannhw/FLaNK-DataFlows/blob/main/jdbc/README.md  Tip:if you have missing core-site / CDP Environmental settings then paste these in to deploy. /home/nifi/additional/secret/env_config/core-site.xml,/home/nifi/additional/secret/env_config/hive-site.xml,/home/nifi/additional/secret/env_config/ssl-client.xmlMay 12, 2023 Attend the Webinar .   Rewatch on demand.Register for the Best in Flow Data Flow Sandbox Environment during the event or after via email you received when signing up.Work on the the tutorials to get familiar.Join my Zoom Sessions on May 4 @1pm EST, May 5 @ 9am EST, May 8 @11am EST, and May 9 @ 9am and more .   I will start more on demand if there is need.Sign up for the slack  group a

Dynamic Allocation in Apache Spark 1. Introduction   In Apache Spark, resource allocation is a critical aspect of optimizing the performance of Spark applications.   Spark provides two mechanisms to allocate resources:  Static Resource Allocation (SRA) Dynamic Resource Allocation (DRA) 1. Static Resource Allocation In static resource allocation, the resources are pre-allocated to the Spark application before it starts running. The amount of resources is fixed and cannot be changed during runtime. It means that if the Spark application requires more resources than what was allocated, it will result in longer execution times or even failure of the job. Static resource allocation is suitable for scenarios where the resource requirements of the Spark application are known in advance and the workload is consistent throughout the job. Disadvantages of Static Resource Allocation Inefficient Resource Utilization: Static resource allocation mode may lead to inefficient

Introduction: Cloudera Data Engineering service is a robust and flexible platform for managing Data Engineering workloads.CDE service allows you to manage workload configurations such as choosing the spark version to use, orchestrate your DE pipelines using Airflow as well as provides a rich API to manage and automate pipelines remotely from the command line. At times, however, one is unable to use the standard CDE configurations for any of the following reasons, The workload requires a very customized set of packages and libraries with version specificity that needs to be baked together and is unique from other workloads in the CDE Service.  Additional  access requirements (e.g. root access)  for package installations are necessary, which are not provided by default CDE runtime libraries Such unique requirements are addressed by building custom runtimes and pushing these runtimes as docker images in an external docker registry. CDE services pull these images from the

IntroductionData Scientists work iteratively in a stochastic space to build models for addressing business use cases. In a typical Machine Learning workflow, Data Scientists iterate by combining their training data with an array of algorithms. In this activity, called model training and evaluation, there are two decisions being made. The first decision involves identifying the dataset for training and testing. Identifying a dataset that is representative of the actual data that the ML system is likely to encounter in a real-life scenario is a key success factor for the system. This helps a trained model to make better predictions when applied to real data. The second decision involves comparing different algorithms around a set of performance metrics. Since both of these decisions tend to be more heuristic-based rather than rule-based, this process is also called experimentation in machine learning parlance.Cloudera AI has simplified experimentation during the model training step of a

In HDP 2.5, Zeppelin Notebook supports the ability to impersonate a user's security context when accessing a data set. This is critical to allow multi-tenant, fine-grained access using Ranger authorization policies. In order to support this, the user will first need to authenticate to the Zeppelin UI.(In Part 2, we harden this configuration to use LDAPS). Zeppelin provides a few different authentication mechanisms (based on what the underlying Apache Shiro project supports); we'll use OpenLDAP in this walkthrough. In this article, we'll assume you already have an OpenLDAP server installed in your environment, which contains the users to which you want to provide access to the Zeppelin UI (I plan to create a supporting article regarding OpenLDAP installation). We'll further assume that these users exist as local OS users (this is outside the scope of this article, but we can use a solution like SSSD for this). Our organizational structure is as follows: With this structure, our

Linux Authentication and Authorization MechanismSecurity plays a very important role in making any software enterprise ready. Linux has a robust security architecture yet it allows plug-gable module to connect to any external Identity manager for authenticating and authorizing users.The two most important modules that play a very important role in providing the security feature at OS level are 1. PAM 2. NSSPAM: The Pluggable Authentication Module allows integration of various authentication technologies such as standard UNIX, RSA, DCE, LDAP etc. into system services such as login, passwd, rlogin, su, ftp, ssh etc. without changing any of these services.First implemented by Sun Solaris, PAM is now the standard authentication framework of many Linux distributions, including RedHat and Debian. It provides an API through which authentication requests are mapped into technology specific actions (implemented in the so called pam modules). This mapping is done by PAM configuration files, i

IntroductionApache Kafka is a high-throughput distributed messaging system that has become one of the most common landing places for data within an organization. Given that Apache NiFi's job is to bring data from wherever it is, to wherever it needs to be, it makes sense that a common use case is to bring data to and from Kafka. The remainder of this post will take a look at some approaches for integrating NiFi and Kafka, and take a deep dive into the specific details regarding NiFi's Kafka support. NiFi as a Producer A common scenario is for NiFi to act as a Kafka producer. With the advent of the Apache MiNiFi sub-project, MiNiFi can bring data from sources directly to a central NiFi instance, which can then deliver data to the appropriate Kafka topic. The major benefit here is being able to bring data to Kafka without writing any code, by simply dragging and dropping a series of processors in NiFi, and being able to visually monitor and control this pipeline.NiFi as a Consumer In som

Display query metrics of Analyzer/Optimizer Rules To display the query metrics of effective runs of Analyzer/Optimizer Rules, we need to use the RuleExecutor object.  RuleExecutor metrics will help us to identify which rule is taking more time. object RuleExecutor { protected val queryExecutionMeter = QueryExecutionMetering() /** Dump statistics about time spent running specific rules. */ def dumpTimeSpent(): String = { queryExecutionMeter.dumpTimeSpent() } /** Resets statistics about time spent running specific rules */ def resetMetrics(): Unit = { queryExecutionMeter.resetMetrics() } def getCurrentMetrics(): QueryExecutionMetrics = { queryExecutionMeter.getMetrics() }} Display the query metrics using Scala code: import org.apache.spark.sql.catalyst.rules.RuleExecutorvar df = spark.range(100).toDF()// Do this first or your values will be cumulativeRuleExecutor.resetMetrics()for (i <- 1 to 500) { df = df.withColumn("id_" + i, col("id") + i)