Resources

According to a survey conducted by Kaggle in 2021, Python is still the most commonly used programming language for data science with over 80% of respondents choosing it as their preferred language. However, R continues to be a popular language among data scientists, with over 15% of respondents choosing it as their primary language. One of the reasons for R's continued popularity is its strong statistical analysis capabilities. R was designed specifically for statistical computing and provides a rich ecosystem of packages for data analysis and visualization. This makes R a powerful tool for data scientists who need to analyze large datasets and perform complex statistical modeling. In this article, we'll delve into how to deploy R models in CML, highlighting the steps and key considerations to keep in mind when building and deploying models in this environment. CMLs Model Framework As a refresher, let's revisit the key concepts of a model in CML. CML's framework all

CODE + COMMUNITY This week in FLaNK Stack Weekly, we have some events going on, a meetup in preparation and a lot of interesting new tools to explore. Please  Join my meetup groups.  We will be hybrid so if you are remote you can still see via zoom or Youtube.   For those in the Princeton, New York City or Philadelphia are we will be in person as well.https://www.meetup.com/futureofdata-princeton/https://www.meetup.com/futureofdata-newyork/https://www.meetup.com/futureofdata-philadelphia I have a meetup in person at our San Francisco office.   I will also be speaking at the Real-Time Analytics Summit that week.https://www.meetup.com/futureofdata-sanfrancisco/events/292453316/ This is Issue #77 and if you wish to look at all of our back issues, check them out in github.   They are in a few different formats.https://github.com/tspannhw/FLiPStackWeekly I travel the world spreading the word of streaming, please join us.https:

Hi, I have been contacted by Cloudera for a job where I sign up for training and then press buttons on a screen to leave positive reviews for various products.  The screen is this: I have been invited to a WhatsApp group but whenever I ask for details to verify that this is a real job the people get very defensive.  I was called paranoid and aggressive because I asked for linkedIn profiles or GlassDoor reviews etc.   I have also been told by my contact that all of the income is tax free and the original person who contacted me has since deleted their Telegram account. Can anyone confirm whether this is a real job, please? 

Special shout-out to @zoram, who provided these instructions.  This article will take you through these steps (executing each step with a screenshot).  This article specifies an edge case that involves processing large amounts of non-tabular datasets such as images.  The data lake is the preferred storage for CML as object storage can scale to billions of objects with tabular data.  If you qualify for this edge case (non-tabular datasets), please delete the dataset from EFS/NFS as soon as possible (after processing), since large amounts of data on EFS/NFS negatively affect backup and recovery times. When it comes to non-tabular data, you may not want to use object storage with standard readers such as Boto3 (for AWS), which has more latency attached to each read.  If we move to a faster I/O such as EFS, which is already used by your CML workspace, we can load data to your existing CML workspace EFS.  It's a simple process that we'll take you

With the latest release of Apache NiFi 1.2.0 the JoltTransformJson Processor became a bit more powerful with an upgrade to the Jolt library (to version 0.1.0) and the introduction of expression language (EL) support. This now provides users the ability to create dynamic specifications for JSON transformation and to perform some data manipulation tasks all within the context of the processor. Internal caching has also been added to improve overall performance. Let’s take an example of transformation Twitter json payload seen below: {"created_at":"Wed Mar 29 02:53:48 +0000 2017","id":846918283102081024,"id_str":"846918283102081024","text":"CSUB falls to Georgia Tech 76-61 in NIT semifinal game. @Bakersfieldcali @BVarsityLive @CSUBAthletics @CSUB_MBB\u2026 https:\/\/t.co\/9e5dQesIbg","display_text_range":[0,140],"source":"\u003ca href=\"http:\/\/twitter.com\" rel=\"nofollow\"\u003eTwitter Web Client\u003c\/a\u003e","truncated":true,"in_reply_to_status_id":null,"in_reply_to_status_id_s

If you've ever wanted to automate CDP, the CDP CLI is the tool to use.  Anything from pulling Virtual Warehouse details to spinning up a new Virtual Warehouse, the CDP CLI is a powerful tool.  Not only Data Warehouse driven, the CDP CLI encompasses all pieces of CDP.     CML is an IDE for all things.  Working on a project from within CML should be a simple and straightforward process.  If you're creating an application that will interact with CDP in one way or another, the CDP CLI can be installed in less than sixty seconds.  Once installed within your CML Project, you'll have the full power of the CDP CLI at your fingertips.     Step 1: Within your session launch the terminal access    Step 2: Within your session Terminal, type pip install cdpcli and click enter   pip install cdpcli    Step 3: Generate an API Access Key (if you don't already have one) https://docs.cloudera.com/cdp-public-cloud

Problem Description When importing a project on CDSW / CML from a private repository in Gitlab / GitHub using the repository URL over HTTPS, it will fail to be created. The URL that is passed here is:      https://github.com/garrynv/demo123.git     Since the repository is private, you cannot only pass the URL to import it. GitHub will not be able to authenticate the request and will reject it. In the early days, users used to pass their username and password in the HTTPS URL and it used to work. something like below:       https://username:password@github.com/garrynv/demo123.git       However, due to restrictions on Gitlab / GitHub end, this will not work anymore. Users will have to generate a PAToken and pass the PAToken to access private repositories remotely. Solution You can refer to the following documents on how to generate Personal Access Token  Github: Creating a personal access token Gitlab: Personal a

Cloudera CSP CE Plus Apache Pulsar See:  pulsar-csp-ce Source:   https://github.com/tspannhw/pulsar-csp-ce Integration https://community.cloudera.com/t5/Cloudera-Stream-Processing-Forum/Using-Apache-Pulsar-with-SQL-Stream-Builder/m-p/349917 Once running in Docker http://localhost:18121/ui/login Login admin/admin SQL Testing CREATE TABLE pulsar_test ( `city` STRING) WITH ( 'connector' = 'pulsar', 'topic' = 'topic82547611', 'value.format' = 'raw', 'service-url' = 'pulsar://Timothys-MBP:6650', 'admin-url' = 'http://Timothys-MBP:8080', 'scan.startup.mode' = 'earliest', 'generic' = 'true');CREATE TABLE `pulsar_table_1670269295` ( `col_str` STRING, `col_int` INT, `col_ts` TIMESTAMP(3), WATERMARK FOR `col_ts` AS col_ts - INTERVAL '5' SECOND) WITH ( 'format' = 'json' -- Data format -- 'json.encode.decimal-as-plain-number' = 'false' -- Optional flag to specify whether to encode all decimals as plain numbers instead of possible scientific notations, false by de

Spark Rolling event log files   1. Introduction While running a long-running spark application (for example streaming application), the spark will generate a larger/huge single event log file until the Spark application is killed or stopped. Maintaining a single event log file which may cost a lot to maintain and also requires a bunch of resources to replay per each update in the Spark History Server.   To avoid creating. a single huge event log file, the spark team created a rolling event log file.   2. Enabling the Spark Rolling Event logs in CDP Step1: Enable the rolling event logs and set the max file size   CM -->Spark 3 --> Configuration --> Spark 3 Client Advanced Configuration Snippet (Safety Valve) for spark3-conf/spark-defaults.conf.  spark.eventLog.rolling.enabled=truespark.eventLog.rolling.maxFileSize=128m  The default spark.eventLog.rolling.maxFileSize value will be 128MB. The minimum value is 10MB.   Ste

A recent update to Cloudera Machine Learning brings the ability to create custom code editors with ML runtimes. This article shows the process of creating and adding an ML Runtime to CML that uses a different editor. First, you will create a Docker image that is configured to use a custom editor, specifically RStudio, and then add it to your workspace.Step 1: Create and upload the Docker ImageNote: If you just want to use RStudio, you can skip this step and use an image that has already been uploaded: peterableda/rstudio-cloudera-runtime:2022.04-8 You will need to have Docker installed and running to do this step. First, clone the repo and use the RStudio 1.4 directory. In a terminal window, run the following commands listed below. $ git clone https://github.com/cloudera/community-ml-runtimes$ cd rstudio_1.4 Now build the Docker image. You need to replace the peterableda/ tag details with the details of the container registry you need to use. The rstu

Missing BlockMark missing if all of the block replicas of that file is not reported to Namenode.Corrupt BlockMark corrupt if all of the block replicas of that file is corrupted (Or) none of them are reported to Namenode.The checklist must be done before you confirm block is corrupted/missing.Check if all datanodes are running in the clusterCheck if you see dead datanodesCheck if disk failure from multiple datanodeCheck if disk out of space from multiple datanodeCheck if block report is rejected by namenode (It can be seen from namenode log as a warning/error)Check if you changed any config groupsCheck if block physically exists in local filesystem or removed by users unknowingly. Ex: "find <dfs.datanode.data.dir> -type f -iname <blkid>*". Repeat the same step in all datanodesCheck if too many blocks hosted in a single datanodeCheck if block report fails with "exceeding max RPC size", default 64 MB. You can see this warning from namenode log "Protocol message was too large.

Throughout my seven years working with Cloudera/Hortonworks, I'm always learning new things.  One thing I've learned from the Cloudera/Hortonoworks merger was how amazing CDSW/CML is as a product.  CML isn't only for Data Scientists, it's for anyone that needs an IDE (Integrated Development Environment).  Coming from a development background working with Eclipse and IntelliJ, you become dependent on a solid IDE. In the past, I've used IDEs to develop applications, and then a build process would eventually deploy to a run environment.  This is where CML shines, you're able to run your applications within CDP at scale with enormous amounts of data.  Anyone using CML is over the moon running their applications/projects within CML because of the simplicity, as I'll demonstrate below.  The tagline of CML is "BYOL" (Bring your own libraries), meaning ALL libraries are welcome (outside of the Cloudera ecosystem).  This differentiates Cloudera from

Have you ever noticed some lingering old rolled log files in your nifi logs directory that never seem to get deleted?This is a by-product of how logback works depending on how you have it configured.-Lets take a look at a default logback.xml configuration from NiFi: <appender name="APP_FILE"> <file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app.log</file> <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy"> <!-- For daily rollover, use 'app_%d.log'. For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'. To GZIP rolled files, replace '.log' with '.log.gz'. To ZIP rolled files, replace '.log' with '.log.zip'. --> <fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern> <maxFileSize>100MB</maxFileSize> <!-- Control

Overview - ExecuteProcess  When developing custom logic as part of our dataflow, having the ability to execute a custom script is a boom especially when it comes to compensating for certain capabilities which are not present out of the box or may not be the need of the hour now for the larger audience. :package:  NiFi makes this possible by providing a processor called ExecuteProcess which enables you to execute custom code. It leverages the underlying operating system to execute the command/process. BackgroundOver the course of using this processor I often had trouble with managing different modules that were required by different ExecuteProcess processors within my flow. At times I was looking at having different versions of the same modules for different set of scripts. Virtual environments in python came to my rescue here. Hence as a best practise it is always advised to use virtual environments to setup additional modules while using Python code within the Exec

If there is a necessity for migrating the existing NFS workspace in CML to a new NFS storage, do the following:  Mount the existing NFS directory and the destination NFS directory and manually copy all the data.[Alternative to Step1] Execute the projectCopy.sh script following the instruction in the README file. (Note: The script, by default, will do only the NFS swap. (script and readme files attached))Modify the NFS mount in the existing helm install:Save the existing helm valueshelm get values mlx-<namespace-name> -n <namespace-name> -o yaml> old.yamlModify the NFSServer and ProjectsExport in the old.yaml file.Example :NFSServer: 10.102.47.132ProjectsExport: /eng-ml-nfs-azure/ageorge→NFSServer: 10.102.47.134ProjectsExport: /ageorge-netapp-volume-v3In my case, I changed NFS server 10.102.47.132:/eng-ml-nfs-azure/ageorge to10.102.47.134:/ageorge-netapp-volume-v3Get the release name from old.yaml:grep GitSHA old.yamlExample:GitSHA: 2.0.35-b1Get the release name

To install Apache NiFi on Macbook, do the following:  Download the Zip file from Apache nifi Downloads. Ensure that you download the latest version zip file Apache NiFi Binary 1.19.1 [OpenPGP] [SHA-256] [SHA-512].  Download it to MBP, and then unzip it: chmod +x bin/nifi.sh Run the following command for User Generate: bin/nifi.sh set-single-user-credentials zzeng <my long password> Run the following command to Start: bin/nifi.sh start bin/nifi.sh status netstat -an | grep LIST Check the  8443 port is LISTEN: tcp4 0 0 127.0.0.1.8443 *.* LISTEN Login from https://localhost:8443/nifi/

Why scale vertically?Once a DL or DH is provisioned with given parameters (instance types, root volume size etc), later there is currently no way in CDP / cloudbreak to change those parameters. However, with time the load may increase and it may not be possible to terminate and re-launch the DL or DH. In this case you might need to scale vertically via Azure console.On the other hand this scaling plays only partially with CDP features, so the long-term fix is rather upgrading to a larger duty size.Overview Both VM and root disk resize steps do require the VM to be in a stopped state. So you can stop the DH or DL and perform both steps.Scaling the virtual machine sizeBased on the azure doc for resizing VMs. After resizing the VM the new size will be in effect until you do a repair or upgrade. After that you will need to do the resize again, if you do not have HA setup.If, however, you have a HA setup, keep in mind to change both VMs: in that way, when the repair changes the pr

Datagen lets you generate data on all services provided by Cloudera (HDFS, Hive, HBase, Ozone, Kafka, Kudu, SolR, Local files), in any format (CSV, JSON, Avro, Parquet, ORC). Users can generate pre-defined data (such as customers or transactions) or create their own data model which supports a wide variety of types. Users can schedule data generation and provide a large set of configuration (such as replication factor, keys etc…).  Datagen is a basic web server exposing APIs that deploys natively as a service in CDP 7.1.7+. Why ?What ?InstallationBasic Data GenerationConfigurationAPICustom Data GenerationResources Why ? There could be many reasons to install and use Datagen, here is a list of some:Test your installationBenchmark your platformReproduce another environment and data (usually a production in a build)Test your code on sample dataTest your code on data at scale before going to productionMake a Proof Of Concept cluster with test dataTest one

OverviewIn this post we will discuss using dbt with the Cloudera Data Platform, and show you how to get started by connecting dbt to your Impala Data Warehouse. You’ll also find links to a dbt example project that you can use to bootstrap your dbt journey.Version Notes:The adapter has been tested on the following version:python: 3.9Cloudera Data Engineering release (1.15-h1)dbt-core: 1.3.0Cloudera Data WarehouseCloudera Data Warehouse (CDW) is a CDP Public Cloud service for self-service creation of independent data warehouses and data marts that autoscale up and down to meet your varying workload demands. The Data Warehouse service provides isolated compute instances for each data warehouse/mart, automatic optimization, and enables you to save costs while meeting SLAs. Both Apache Impala and Apache Hive are available through Cloudera Data Warehouse.What is dbt?dbt is quickly gaining popularity as a key component of the modern data stack; a tool that enables the creation of data pipelin

IntroductionUse of data in Excel spreadsheets is commonplace in the enterprise and often results in widespread yet highly manual and loosely defined dataflows. The manual and loose nature of Excel data flows are at odds with the automated way we expect data to move through the enterprise. As a result, these Excel data flows lack the accuracy, speed and transparency leveraged by automated flows. Shouldn’t Excel data flows be automated too?In this article I show how to use NiFi to automate the reading, writing and validation of Excel contents (using the full capability set humans use) and how to fit this into a larger workflow. I focus here on building an Excel client to operate on Excel data during your workflow. Designing your own Excel workflows depends on how you do things manually today and on your ability to innovate new ways to leverage Excel data in your larger data landscape and initiatives.TechnologiesThe key technology for reading and writing to Excel is Apache’s POI Java

 In this article, we will walk through the steps to integrate Data Engineering Experience & Operational DataBase Experience within a few clicks with an example of reading & writing data into Phoenix tables using PySpark script.  As you know Cloudera Operational Database supports both Traditional relational schema and Evolutionary schema with the capability to transact on petabytes of structured and unstructured data generated by businesses today further details are attached. Whereas Cloudera Data Engineering streamlines and operationalizes data pipelines securely at any scale to increase efficiency and accelerate time to value further details attached.Example: Customer 360 - Cloudera Operational Database will store customer profiling data for business applications and the data is modeled & prepared with Cloudera Data Engineering. Both capabilities come with High Availability, Scalability, Mission Critical Performance, and Fl

OverviewCloudera has implemented dbt adapters for Hive, Spark, and Impala. In addition to providing the adapters, Cloudera is also offering dbt capabilities to our on-prem customers (CDP PvC-Base). Our solution satisfies our customers' stringent security and privacy requirements while providing an easy-to-use turnkey solution for practitioners.In this document, we will show how users can run dbt on YARN without having to worry about anything else. We have packaged everything as a .tar.gz file.If you are new to dbt, refer to the article to learn more about dbt. PrerequisitesThe cluster should have Python 3.7+The user should have the ability to ssh into a gateway machine The user should have access to the git repo with the dbt models and has the ability to create PRs in that git repo with changes. If you are creating your own repo as an analyst, refer to this Getting started with dbt Core The user SSH key can be configured for access to the git repoCompatibility>= CentO

IntroductionCloudera recently announced the open-source dbt adapters for all the engines in Cloudera Data Platform (CDP)—, Apache Impala, and Apache Spark, with added support for Apache Livy and Cloudera Data Engineering.  In addition to providing the adapters, Cloudera is offering a turn-key solution to be able to manage the end-to-end software development life cycle (SDLC) of dbt models. This solution is available on all clouds, as well as on-prem deployments of CDP.  In this article, we show how our customer data teams can streamline their data transformation pipelines in the Cloudera Data Platform and deliver high-quality data that their business can trust. Our solution satisfies the stringent security and privacy requirements of our customers while providing an easy-to-use turnkey solution for practitioners.dbt’s end-to-end software development life cycleA key advantage of using dbt is that it provides a framework for analysts to easily follow software engineer

Installing and configuring Livy on CDH 6.x.x Livy is a preferred way to run Spark jobs on several Hadoop installations, but not on CDH. While preparing for a CDP migration, one of our use-cases switched to Apache Airflow to run jobs without requiring an edge node or "bastion node" and they wanted to begin using Airflow before the CDP migration, so they asked me to install Livy on a CDH edge node.   A search online for Livy on CDH returned little helpful information, but I did find information on how to download and install it at https://livy.apache.org/ Step 1: Determine which account will be used to run Livy Linux security will allow an application to access or execute any program or file the executing account can access unless you configure selinux or another access management software. Pick an account to run Livy from. Step 2: Set up a keytab You'll need a Kerberos principal. If you use Active Directory principals with your CDH deployment, then this account will be outsid

Hi All, When we use MySQL 8 for HUE DB we get the below error  Unexpected error. Unable to verify the database connectionIn Cloudera manager server logs :  2022-11-04 09:18:56,152 INFO scm-web-5325:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult, Status:2002022-11-04 09:18:56,575 INFO CommandPusher-1:com.cloudera.server.cmf.CommandPusherThread: Acquired lease lock on DbCommand:5472022-11-04 09:18:56,578 INFO CommandPusher-1:com.cloudera.cmf.service.AbstractOneOffHostCommand: Unsuccessful 'HueTestDatabaseConnection'2022-11-04 09:18:56,578 INFO CommandPusher-1:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: Command exited with code: 12022-11-04 09:18:56,578 INFO CommandPusher-1:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: self.ensure_connection()File "/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.