Community Articles

NiFi - Oracle Connect Processor: ExecuteSQL 1) Prepare Oracle Data (Optional) Prepare the data, and create the Oracle table:   CREATE TABLE USER_MASTER( USER_ID VARCHAR2(8) NOT NULL, DEPT_NO VARCHAR2(8), USER_NAME VARCHAR2(32), CREATED_ON DATE DEFAULT SYSDATE, MODIFIED_ON DATE, CONSTRAINT pk_USER_MASTER PRIMARY KEY(USER_ID));   Insert the data into Oracle Table:   INSERT INTO USER_MASTER (USER_NAME, DEPT_NO, USER_ID) VALUES ('Hayashi　1 Ro','1001','1');INSERT INTO USER_MASTER (USER_NAME, DEPT_NO, USER_ID) VALUES ('Hayashi　2 Ro','1001','2');INSERT INTO USER_MASTER (USER_NAME, DEPT_NO, USER_ID) VALUES ('Hayashi　3 Ro','1001','3');INSERT INTO USER_MASTER (USER_NAME, DEPT_NO, USER_ID) VALUES ('Hayashi　4 Ro','1001','4');INSERT INTO USER_MASTER (USER_NAME, DEPT_NO, USER_ID) VALUES ('Hayashi　5 Ro','1001','5');INSERT INTO USER_MASTER (USER_NAME, DEPT_NO, USER_ID) VALUES ('Hayashi　6 Ro','1001','6');INSERT INTO USER_MASTER (USER_NAME

In a recent release, CDW is now fully integrated with RAZ.  This means once you provision an environment in CDP with RAZ enabled, adding an S3 bucket into CDW follows the same principles as the services within CDP (CML, CDE, COD, CDF, and DataHub).   I was able to add a new S3 bucket called "ryancicak" in two steps, after receiving the 403 Forbidden error within Hue: Two steps:   Step 1 - Since I used the -datalake-admin-role for RAZ, I modified the "datalake-admin-s3-policy to include my new bucket "ryancicak" with the two lines: "arn:aws:s3:::ryancicak","arn:aws:s3:::ryancicak/*"   *Notice I also removed the subdirectory from my cicak-cdp-stack to include the root directory.  This is important to make this modification as well.  Instead of "cicak-cdp-stack/my-data", I state "cicak-cdp-stack" Step 2 - I added a new policy in Ranger under cm_s3 for my bucket "ryancicak", for the users rcicak (me), and Hive.     DONE - When re-r

The Phoenix documentation here leaves out a few pieces in order to make a successful connection to HBase, through the Phoenix Driver. They assume that the connection is from the 'localhost'. May work great, but that's unlikely in the real world.Required Jarsphoenix-client.jar hbase-client.jar (not mentioned in the Phoenix Docs)URL DetailsThe full adbc connection URL syntax for phoenix is:Basic Connections jdbc:phoneix[:zk_quorum][:zk_port][:zk_hbase_path]The "zk_quorum" is a comma separated list of the ZooKeeper Servers.The "zk_port" is the ZooKeeper port.The "zk_hbase_path" is the path used by Hbase to stop information about the instance. On a 'non' kerberized cluster the default zk_hbase_path for HDP is '/hbase-unsecure'.Sample JDBC URLjdbc:phoenix:m1.hdp.local,m2.hdp.local,d1.hdp.local:2181:/hbase-unsecure For Kerberos Cluster Connections jdbc:phoneix[:zk_quorum][:zk_port][:zk_hbase_path][:headless_keytab_file:principal] Sample JDBC URL jdbc:phoenix:m1.hdp.local,m2.hdp.local,d1.h

Cloudera DataFlow Connectors We would love to have you and your fellow colleagues attend this session. Please feel free to share this internally. We look forward to having you join us!Reading from Iceberghttps://github.com/tspannhw/FLaNK-DataFlows/blob/main/jdbc/README.md  Tip:if you have missing core-site / CDP Environmental settings then paste these in to deploy. /home/nifi/additional/secret/env_config/core-site.xml,/home/nifi/additional/secret/env_config/hive-site.xml,/home/nifi/additional/secret/env_config/ssl-client.xmlMay 12, 2023 Attend the Webinar .   Rewatch on demand.Register for the Best in Flow Data Flow Sandbox Environment during the event or after via email you received when signing up.Work on the the tutorials to get familiar.Join my Zoom Sessions on May 4 @1pm EST, May 5 @ 9am EST, May 8 @11am EST, and May 9 @ 9am and more .   I will start more on demand if there is need.Sign up for the slack  group a

Dynamic Allocation in Apache Spark 1. Introduction   In Apache Spark, resource allocation is a critical aspect of optimizing the performance of Spark applications.   Spark provides two mechanisms to allocate resources:  Static Resource Allocation (SRA) Dynamic Resource Allocation (DRA) 1. Static Resource Allocation In static resource allocation, the resources are pre-allocated to the Spark application before it starts running. The amount of resources is fixed and cannot be changed during runtime. It means that if the Spark application requires more resources than what was allocated, it will result in longer execution times or even failure of the job. Static resource allocation is suitable for scenarios where the resource requirements of the Spark application are known in advance and the workload is consistent throughout the job. Disadvantages of Static Resource Allocation Inefficient Resource Utilization: Static resource allocation mode may lead to inefficient

Introduction: Cloudera Data Engineering service is a robust and flexible platform for managing Data Engineering workloads.CDE service allows you to manage workload configurations such as choosing the spark version to use, orchestrate your DE pipelines using Airflow as well as provides a rich API to manage and automate pipelines remotely from the command line. At times, however, one is unable to use the standard CDE configurations for any of the following reasons, The workload requires a very customized set of packages and libraries with version specificity that needs to be baked together and is unique from other workloads in the CDE Service.  Additional  access requirements (e.g. root access)  for package installations are necessary, which are not provided by default CDE runtime libraries Such unique requirements are addressed by building custom runtimes and pushing these runtimes as docker images in an external docker registry. CDE services pull these images from the

IntroductionData Scientists work iteratively in a stochastic space to build models for addressing business use cases. In a typical Machine Learning workflow, Data Scientists iterate by combining their training data with an array of algorithms. In this activity, called model training and evaluation, there are two decisions being made. The first decision involves identifying the dataset for training and testing. Identifying a dataset that is representative of the actual data that the ML system is likely to encounter in a real-life scenario is a key success factor for the system. This helps a trained model to make better predictions when applied to real data. The second decision involves comparing different algorithms around a set of performance metrics. Since both of these decisions tend to be more heuristic-based rather than rule-based, this process is also called experimentation in machine learning parlance.Cloudera AI has simplified experimentation during the model training step of a

In HDP 2.5, Zeppelin Notebook supports the ability to impersonate a user's security context when accessing a data set. This is critical to allow multi-tenant, fine-grained access using Ranger authorization policies. In order to support this, the user will first need to authenticate to the Zeppelin UI.(In Part 2, we harden this configuration to use LDAPS). Zeppelin provides a few different authentication mechanisms (based on what the underlying Apache Shiro project supports); we'll use OpenLDAP in this walkthrough. In this article, we'll assume you already have an OpenLDAP server installed in your environment, which contains the users to which you want to provide access to the Zeppelin UI (I plan to create a supporting article regarding OpenLDAP installation). We'll further assume that these users exist as local OS users (this is outside the scope of this article, but we can use a solution like SSSD for this). Our organizational structure is as follows: With this structure, our

Linux Authentication and Authorization MechanismSecurity plays a very important role in making any software enterprise ready. Linux has a robust security architecture yet it allows plug-gable module to connect to any external Identity manager for authenticating and authorizing users.The two most important modules that play a very important role in providing the security feature at OS level are 1. PAM 2. NSSPAM: The Pluggable Authentication Module allows integration of various authentication technologies such as standard UNIX, RSA, DCE, LDAP etc. into system services such as login, passwd, rlogin, su, ftp, ssh etc. without changing any of these services.First implemented by Sun Solaris, PAM is now the standard authentication framework of many Linux distributions, including RedHat and Debian. It provides an API through which authentication requests are mapped into technology specific actions (implemented in the so called pam modules). This mapping is done by PAM configuration files, i

IntroductionApache Kafka is a high-throughput distributed messaging system that has become one of the most common landing places for data within an organization. Given that Apache NiFi's job is to bring data from wherever it is, to wherever it needs to be, it makes sense that a common use case is to bring data to and from Kafka. The remainder of this post will take a look at some approaches for integrating NiFi and Kafka, and take a deep dive into the specific details regarding NiFi's Kafka support. NiFi as a Producer A common scenario is for NiFi to act as a Kafka producer. With the advent of the Apache MiNiFi sub-project, MiNiFi can bring data from sources directly to a central NiFi instance, which can then deliver data to the appropriate Kafka topic. The major benefit here is being able to bring data to Kafka without writing any code, by simply dragging and dropping a series of processors in NiFi, and being able to visually monitor and control this pipeline.NiFi as a Consumer In som

Display query metrics of Analyzer/Optimizer Rules To display the query metrics of effective runs of Analyzer/Optimizer Rules, we need to use the RuleExecutor object.  RuleExecutor metrics will help us to identify which rule is taking more time. object RuleExecutor { protected val queryExecutionMeter = QueryExecutionMetering() /** Dump statistics about time spent running specific rules. */ def dumpTimeSpent(): String = { queryExecutionMeter.dumpTimeSpent() } /** Resets statistics about time spent running specific rules */ def resetMetrics(): Unit = { queryExecutionMeter.resetMetrics() } def getCurrentMetrics(): QueryExecutionMetrics = { queryExecutionMeter.getMetrics() }} Display the query metrics using Scala code: import org.apache.spark.sql.catalyst.rules.RuleExecutorvar df = spark.range(100).toDF()// Do this first or your values will be cumulativeRuleExecutor.resetMetrics()for (i <- 1 to 500) { df = df.withColumn("id_" + i, col("id") + i)

According to a survey conducted by Kaggle in 2021, Python is still the most commonly used programming language for data science with over 80% of respondents choosing it as their preferred language. However, R continues to be a popular language among data scientists, with over 15% of respondents choosing it as their primary language. One of the reasons for R's continued popularity is its strong statistical analysis capabilities. R was designed specifically for statistical computing and provides a rich ecosystem of packages for data analysis and visualization. This makes R a powerful tool for data scientists who need to analyze large datasets and perform complex statistical modeling. In this article, we'll delve into how to deploy R models in CML, highlighting the steps and key considerations to keep in mind when building and deploying models in this environment. CMLs Model Framework As a refresher, let's revisit the key concepts of a model in CML. CML's framework all

CODE + COMMUNITY This week in FLaNK Stack Weekly, we have some events going on, a meetup in preparation and a lot of interesting new tools to explore. Please  Join my meetup groups.  We will be hybrid so if you are remote you can still see via zoom or Youtube.   For those in the Princeton, New York City or Philadelphia are we will be in person as well.https://www.meetup.com/futureofdata-princeton/https://www.meetup.com/futureofdata-newyork/https://www.meetup.com/futureofdata-philadelphia I have a meetup in person at our San Francisco office.   I will also be speaking at the Real-Time Analytics Summit that week.https://www.meetup.com/futureofdata-sanfrancisco/events/292453316/ This is Issue #77 and if you wish to look at all of our back issues, check them out in github.   They are in a few different formats.https://github.com/tspannhw/FLiPStackWeekly I travel the world spreading the word of streaming, please join us.https:

Special shout-out to @zoram, who provided these instructions.  This article will take you through these steps (executing each step with a screenshot).  This article specifies an edge case that involves processing large amounts of non-tabular datasets such as images.  The data lake is the preferred storage for CML as object storage can scale to billions of objects with tabular data.  If you qualify for this edge case (non-tabular datasets), please delete the dataset from EFS/NFS as soon as possible (after processing), since large amounts of data on EFS/NFS negatively affect backup and recovery times. When it comes to non-tabular data, you may not want to use object storage with standard readers such as Boto3 (for AWS), which has more latency attached to each read.  If we move to a faster I/O such as EFS, which is already used by your CML workspace, we can load data to your existing CML workspace EFS.  It's a simple process that we'll take you

With the latest release of Apache NiFi 1.2.0 the JoltTransformJson Processor became a bit more powerful with an upgrade to the Jolt library (to version 0.1.0) and the introduction of expression language (EL) support. This now provides users the ability to create dynamic specifications for JSON transformation and to perform some data manipulation tasks all within the context of the processor. Internal caching has also been added to improve overall performance. Let’s take an example of transformation Twitter json payload seen below: {"created_at":"Wed Mar 29 02:53:48 +0000 2017","id":846918283102081024,"id_str":"846918283102081024","text":"CSUB falls to Georgia Tech 76-61 in NIT semifinal game. @Bakersfieldcali @BVarsityLive @CSUBAthletics @CSUB_MBB\u2026 https:\/\/t.co\/9e5dQesIbg","display_text_range":[0,140],"source":"\u003ca href=\"http:\/\/twitter.com\" rel=\"nofollow\"\u003eTwitter Web Client\u003c\/a\u003e","truncated":true,"in_reply_to_status_id":null,"in_reply_to_status_id_s

If you've ever wanted to automate CDP, the CDP CLI is the tool to use.  Anything from pulling Virtual Warehouse details to spinning up a new Virtual Warehouse, the CDP CLI is a powerful tool.  Not only Data Warehouse driven, the CDP CLI encompasses all pieces of CDP.     CML is an IDE for all things.  Working on a project from within CML should be a simple and straightforward process.  If you're creating an application that will interact with CDP in one way or another, the CDP CLI can be installed in less than sixty seconds.  Once installed within your CML Project, you'll have the full power of the CDP CLI at your fingertips.     Step 1: Within your session launch the terminal access    Step 2: Within your session Terminal, type pip install cdpcli and click enter   pip install cdpcli    Step 3: Generate an API Access Key (if you don't already have one) https://docs.cloudera.com/cdp-public-cloud

Problem Description When importing a project on CDSW / CML from a private repository in Gitlab / GitHub using the repository URL over HTTPS, it will fail to be created. The URL that is passed here is:      https://github.com/garrynv/demo123.git     Since the repository is private, you cannot only pass the URL to import it. GitHub will not be able to authenticate the request and will reject it. In the early days, users used to pass their username and password in the HTTPS URL and it used to work. something like below:       https://username:password@github.com/garrynv/demo123.git       However, due to restrictions on Gitlab / GitHub end, this will not work anymore. Users will have to generate a PAToken and pass the PAToken to access private repositories remotely. Solution You can refer to the following documents on how to generate Personal Access Token  Github: Creating a personal access token Gitlab: Personal a

Cloudera CSP CE Plus Apache Pulsar See:  pulsar-csp-ce Source:   https://github.com/tspannhw/pulsar-csp-ce Integration https://community.cloudera.com/t5/Cloudera-Stream-Processing-Forum/Using-Apache-Pulsar-with-SQL-Stream-Builder/m-p/349917 Once running in Docker http://localhost:18121/ui/login Login admin/admin SQL Testing CREATE TABLE pulsar_test ( `city` STRING) WITH ( 'connector' = 'pulsar', 'topic' = 'topic82547611', 'value.format' = 'raw', 'service-url' = 'pulsar://Timothys-MBP:6650', 'admin-url' = 'http://Timothys-MBP:8080', 'scan.startup.mode' = 'earliest', 'generic' = 'true');CREATE TABLE `pulsar_table_1670269295` ( `col_str` STRING, `col_int` INT, `col_ts` TIMESTAMP(3), WATERMARK FOR `col_ts` AS col_ts - INTERVAL '5' SECOND) WITH ( 'format' = 'json' -- Data format -- 'json.encode.decimal-as-plain-number' = 'false' -- Optional flag to specify whether to encode all decimals as plain numbers instead of possible scientific notations, false by de

Spark Rolling event log files   1. Introduction While running a long-running spark application (for example streaming application), the spark will generate a larger/huge single event log file until the Spark application is killed or stopped. Maintaining a single event log file which may cost a lot to maintain and also requires a bunch of resources to replay per each update in the Spark History Server.   To avoid creating. a single huge event log file, the spark team created a rolling event log file.   2. Enabling the Spark Rolling Event logs in CDP Step1: Enable the rolling event logs and set the max file size   CM -->Spark 3 --> Configuration --> Spark 3 Client Advanced Configuration Snippet (Safety Valve) for spark3-conf/spark-defaults.conf.  spark.eventLog.rolling.enabled=truespark.eventLog.rolling.maxFileSize=128m  The default spark.eventLog.rolling.maxFileSize value will be 128MB. The minimum value is 10MB.   Ste

A recent update to Cloudera Machine Learning brings the ability to create custom code editors with ML runtimes. This article shows the process of creating and adding an ML Runtime to CML that uses a different editor. First, you will create a Docker image that is configured to use a custom editor, specifically RStudio, and then add it to your workspace.Step 1: Create and upload the Docker ImageNote: If you just want to use RStudio, you can skip this step and use an image that has already been uploaded: peterableda/rstudio-cloudera-runtime:2022.04-8 You will need to have Docker installed and running to do this step. First, clone the repo and use the RStudio 1.4 directory. In a terminal window, run the following commands listed below. $ git clone https://github.com/cloudera/community-ml-runtimes$ cd rstudio_1.4 Now build the Docker image. You need to replace the peterableda/ tag details with the details of the container registry you need to use. The rstu

Missing BlockMark missing if all of the block replicas of that file is not reported to Namenode.Corrupt BlockMark corrupt if all of the block replicas of that file is corrupted (Or) none of them are reported to Namenode.The checklist must be done before you confirm block is corrupted/missing.Check if all datanodes are running in the clusterCheck if you see dead datanodesCheck if disk failure from multiple datanodeCheck if disk out of space from multiple datanodeCheck if block report is rejected by namenode (It can be seen from namenode log as a warning/error)Check if you changed any config groupsCheck if block physically exists in local filesystem or removed by users unknowingly. Ex: "find <dfs.datanode.data.dir> -type f -iname <blkid>*". Repeat the same step in all datanodesCheck if too many blocks hosted in a single datanodeCheck if block report fails with "exceeding max RPC size", default 64 MB. You can see this warning from namenode log "Protocol message was too large.

Throughout my seven years working with Cloudera/Hortonworks, I'm always learning new things.  One thing I've learned from the Cloudera/Hortonoworks merger was how amazing CDSW/CML is as a product.  CML isn't only for Data Scientists, it's for anyone that needs an IDE (Integrated Development Environment).  Coming from a development background working with Eclipse and IntelliJ, you become dependent on a solid IDE. In the past, I've used IDEs to develop applications, and then a build process would eventually deploy to a run environment.  This is where CML shines, you're able to run your applications within CDP at scale with enormous amounts of data.  Anyone using CML is over the moon running their applications/projects within CML because of the simplicity, as I'll demonstrate below.  The tagline of CML is "BYOL" (Bring your own libraries), meaning ALL libraries are welcome (outside of the Cloudera ecosystem).  This differentiates Cloudera from

Have you ever noticed some lingering old rolled log files in your nifi logs directory that never seem to get deleted?This is a by-product of how logback works depending on how you have it configured.-Lets take a look at a default logback.xml configuration from NiFi: <appender name="APP_FILE"> <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file> <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy"> <!-- For daily rollover, use 'app_%d.log'. For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'. To GZIP rolled files, replace '.log' with '.log.gz'. To ZIP rolled files, replace '.log' with '.log.zip'. --> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern> <maxFileSize>100MB</maxFileSize> <!-- Control

Overview - ExecuteProcess  When developing custom logic as part of our dataflow, having the ability to execute a custom script is a boom especially when it comes to compensating for certain capabilities which are not present out of the box or may not be the need of the hour now for the larger audience. :package:  NiFi makes this possible by providing a processor called ExecuteProcess which enables you to execute custom code. It leverages the underlying operating system to execute the command/process. BackgroundOver the course of using this processor I often had trouble with managing different modules that were required by different ExecuteProcess processors within my flow. At times I was looking at having different versions of the same modules for different set of scripts. Virtual environments in python came to my rescue here. Hence as a best practise it is always advised to use virtual environments to setup additional modules while using Python code within the Exec

If there is a necessity for migrating the existing NFS workspace in CML to a new NFS storage, do the following:  Mount the existing NFS directory and the destination NFS directory and manually copy all the data.[Alternative to Step1] Execute the projectCopy.sh script following the instruction in the README file. (Note: The script, by default, will do only the NFS swap. (script and readme files attached))Modify the NFS mount in the existing helm install:Save the existing helm valueshelm get values mlx-<namespace-name> -n <namespace-name> -o yaml> old.yamlModify the NFSServer and ProjectsExport in the old.yaml file.Example :NFSServer: 10.102.47.132ProjectsExport: /eng-ml-nfs-azure/ageorge→NFSServer: 10.102.47.134ProjectsExport: /ageorge-netapp-volume-v3In my case, I changed NFS server 10.102.47.132:/eng-ml-nfs-azure/ageorge to10.102.47.134:/ageorge-netapp-volume-v3Get the release name from old.yaml:grep GitSHA old.yamlExample:GitSHA: 2.0.35-b1Get the release name