Community Articles

The following steps can be used to configure Intellij to run Spark jobs on Cloudera Data Engineering experience (CDE). This way the developers will be able to test their jobs on CDE without having to leave the IDE. The following steps are shown for an Intellij installed on a Windows PC. Complete setting up CDE CLI with Git Bash as per this article. In IntelliJ, go to File > Settings, modify the "shell path" to the following:C:\Program Files\Git\bin\bash.exe" --login -i Test this setting by launching a terminal from within the IntelliJ project and running CDE CLI commands. If you are unable to run CDE CLI commands (with environment variable CDE_CONFIG not configured), you will have the option to setup/override it in the next step. Add a Run/Debug configuration with a new Shell script with the following details. Ignore the error stating that the Shell script is not found. You have the option of overriding the CDE_CONFIG environment variable (in case you want to submit

Download the Cloudera Data Engineering CLI from the Cloudera Data Engineering > CDE Services > Virtual Clusters. Click the Cluster details page, and click the CLI TOOL button to download the CDE CLI. Move the CLI to the desired directory. Here, it is moved to D:\Users\Vijay.Karthikeyan\Documents\utils\cde. Add this directory to PATH, as shown as follows: Create the directory ~/AppData/Local/cloudera/cde to store the CDE configuration files. Make a note of your workload username from the profile page of the CDP control plane, and set workload password for the environment if not done already. Create a file containing the workload password Create the certificate file ca.pem holding the root certificates. Get the certificates here.  Concatenate them and save them as ~/AppData/Local/cloudera/cde/ca.pem. Get the virtual cluster endpoint (This will be entered as the value vcluster-endpoint in the config.yaml file) Create a file config.yaml with the fo

Objective This tutorial consists of two articles. The first walks you through a NiFI flow that utilizes the ValidateRecord processor and Record Reader/Writer controller services to: Convert a CVS file into JSON format Validate the data against a given schema Write the JSON data to either a 'valid' relationship or 'invalid' relationship The second article, modifies the flow to demonstrated the effects of enabling/disabling the "Strict Type Checking" property of the ValidateRecord processor. Note: The ValidateRecord processor was introduced in NiFi 1.4.0.Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.4.0ValidateRecord FlowSupport Files Here is a template of the flow discussed in this tutorial: validaterecord.xml In the spirit of Halloween last week, let's ingest some data that ranks the most popular candies: candy-data-dirty.txt (Change the extension from .txt to .csv after downloading) Note: The CSV dat

New Features of Apache NiFi 1.13.0 Check it out:   https://twitter.com/pvillard31/status/1361569608327716867?s=27 Download today: Apache NiFi Downloads Release Notes: Release Notes (Apache NiFi) Migration:  Migration Guidance New Features ListenFTP UpdateHiveTable - Hive DDL changes - Hive Update Schema i.e. Data Drift i.e. Hive Schema Migration!!!! SampleRecord - different sampling approaches to records (Interval Sampling, Probabilistic Sampling, Reservoir Sampling) CDC updates Kudu updates AMQP and MQTT integration upgrades ConsumeMQTT - readers, and writers added HTTP access to NiFi by default is now configured to accept connections to 127.0.0.1/localhost only. If you want to allow broader access for some reason for HTTP, and you understand the security implications, you can still control that as always by changing the 'nifi.web.http.host' property in nifi.properties as always. That said, take the time to configure proper HTTPS. We offer

In this article, we will learn how to build Twitter NiFi template in a flow management datahub. Pre-Requisite  CDP environment is created and flow management datahub is installed. In that, the required users are added. Steps Login to NiFi by clicking on the NiFi log as shown below: Add GetTwitter process group and put a file from the process group as follows: Configure Twitter process group by right-clicking on GetTwitter process group, and then select Configure; add required parameters: Consumer Key, Consumer Secret, Access Token, Access Token Secret as shown as follows: Configure the PUT file process to store the Twitter JSON files in one of the NiFi hosts.To verify the data received, log in to one of the NiFi hosts, and check the directory mentioned in the above step. 

Running 'pyspark' applications in CML for model generation and prediction, with data residing in COD With the recent addition of Cloudera Operation Database Experience to the CDP Public Cloud, we want to explore how it can be leveraged in the real-life 'DataFlow' end-user scenario. This article talks about how to execute Spark/pyspark job in CML to run modeling task using the data residing in COD. We read the table present in COD and also write back the score table back to the COD once the prediction is done. Getting Started CDP Runtime (supporting COD) >=7.2.2 We assume that CDP environment, datalake, datahub (Data Engineering) have been provisioned. We further assume that experiences COD and CML have been provisioned for the CDP target environment. Note: Please refer to The world’s first enterprise data cloud, if you are just starting with CDP, and get to know how all the requirements can be in place with ease.  Some of the following steps are already documen

It's not a simple process, but one that can be easily done by anyone with a little database admin experience. You have to add a user by restarting cbd with new user and password in Profile. Then you go into the database and move the encrypted password from the new user to the old. This will change the password and leave your old user with access to the clusters and resource you've already built.OverviewEdit the Profile and change both the UAA_DEFAULT_USER_PW to the desired new password and UAA_DEFAULT_USER_EMAIL to a different address. cbd restart – This will add a 2nd user with the new password to the database docker exec -ti cbreak_commondb_1 bash – Starts a bash shell in the database container pg_dump -Fc -U postgres uaadb > uaadb.dump – Makes a backup of the user database psql -U postgres – Starts a postgres shell postgres=# \c uaadb; – Connects to the user database in the postgres shell uaadb=# select * from users; – Shows the two accounts and their encrypted passwordsupdate u

Cloudera Machine Learning provides a number of methods of connecting to other CDP services and experiences such as a Cloudera Data Warehouse. In this post, we will connect using Python and the Impyla library, as well as using the embedded Cloudera Data Visualization. Using Impyla  Within Cloudera Machine Learning, create a new project and set the language to Python 3.6. The connection details are available from the Data Warehouse console by copying the JDBC connection details which will look like.jdbc:impala://coordinator-aws-2-impala-prod.env-j2ln9x.dw.ylcu-atmi.cloudera.site:443/default;AuthMech=3;transportMode=http;httpPath=cliservice;ssl=1;UID=<workload username>;PWD=<workload password> Use the following Python code to install Impyla and configure a connection:!pip3 install impyla==0.16a3USERNAME='<workload username>'IMPALA_HOST='coordinator-aws-2-impala-prod.env-j2ln9x.dw.ylcu-atmi.cloudera.site'IMPALA_PORT='443'from impala.dbapi import connectconn =

Here is a fun one: how do you connect from Python in Cloudera Machine Learning to our Kafka datahub cluster?The documentation is pretty thorough, but it does not have an example of a python client. That's what I'm going to highlight in this article. The good news is that since CML and Datahub run in the same network, you don't need to worry about opening the broker ports, therefore you just need to follow these steps:Step 1: Get and upload your freeIPA certificateStep 2: Find your broker hostnamesStep 3: Setup your clientStep 1: Get and upload your freeIPA certificateGo to your management console > your environment > Actions > Get FreeIPA Certificate:Once downloaded, go to your CML workspace, and upload your file (e.g. /home/cdsw/ca.crt).Step 2: Find your broker hostnamesFor this, go to your Kafka Datahub Cluster > CM UI > Kafka > Instances; you can find the broker hosts here:Step 3: Setup your clientThen, open a session in CML, and use the following paramet

Cloudera Machine Learning provides support for Python3. It is very straightforward to connect a session with an operational database. Provision an Operational Database Log into a CDP instance Select Operational Database Select Create Database Choose the Cloud environment Provide a unique name for the database Click Create Database Once the database has started, make a copy of the Phoenix (Thin) JDBC URL. This will be used as the connection string. Create a Machine Learning Project Within your Cloudera Machine Learning (CML) workspace, create a new project. Provide a name, and choose a blank initial setup. Create a session, and install phoenixdb using the command:!pip3 install phoenixdb Create a new Python file and paste the following code into the notebook.Import the required dependenciesimport phoenixdbimport ioimport json Setup the parameters required to establish the connection with ODB. Refer to the Thin client details. opts = {}opts['authentication'] = 'BASIC'op

In order to convert JSON to XML, you can use ConvertRecord processor available as part of the NiFi tool (I worked on version: 1.11.4). Following are the steps to configure ConvertRecord processor to convert JSON input to XML output: Add a processor – ConvertRecord and right-click to “configure”  Click the Record Reader value and select Create New Service. Select JSONTreeReader and click CREATE.  Similarly, click on the Record Writer value and select Create New Service. Select XMLRecordSetWriter and click CREATE. Final configuration of ConvertRecord should look like this: Click à to configure JSONTreeReader to configure details about the schema that should be read by the registry. The controller service form should open: Click configure of JSONTreeReader to start configuring the JSON Reader and configure as per the following screenshot: When you select Create New Service in the above screenshot, another form opens up. Fill it up ba

Introduction When running Apache Kafka at scale, sooner or later you will start worrying about available disk space, growth rates of topics, or general disk usage. This is especially the case when handling with external producers - outside of your direct control and without configuring hard size limits within your retention configuration. The kafka-log-dirs tool allows you to determine the current size and location of all available partitions on your Kafka cluster. Due to its nature, this tool only represents a snapshot of the current state without any history - and it is far from being convenient to use. You want to know how far? Here's a snippet of the output:   But we want to turn the JSON above into something more shiny and usable, like this:   So, let us figure out how we can facilitate Apache NiFi, Apache Kudu, and Tableau to retrieve consumable visualization and actionable insights about your Kafka cluster! Architecture On a high level, we are going to use Apache NiF

This video provides instructions on how to SSH keys in CDP Public Cloud data hub clusters.It touches on how to generate a new SSH key pair and steps through the process of deploying it for a workload user through the Cloudera Management Console Web UI, as well as using the CDP command-line tool. It discusses the security implications of using the Cloudbreak user for login purposes on data hub hosts and explains why workload user credentials should be used instead in most cases. It also demonstrates using the deployed SSH keys for login to data hub hosts.

Recently, the weekly number of COVID-19 cases in The Netherlands has been steadily dropping week over week. However, underneath this lies a hidden positive trend of cases with the British COVID variant.  In this article, I explain how I made my first visual in Cloudera Data Visualization with just a few clicks. Step 1: The Data The data is created by combining several sources that shed light on the percentage of new infections that are made up with this variant, as well as the total number of cases. As we are here for tech, more than science, I will only highlight the official report of cases per week and an official article that contains the latest percentage, as well as an earlier percentage. Additional explanation on the numbers is given in the post below this article.   From here I could have chosen from many data sources, including a CSV file, but for reproducibility, I decided to upload it to Hive with a simple query:   create table covid_cases asSELECT 431 as b

RAPIDS.ai is a set of Nvidia libraries designed to help data scientists and engineers to leverage the Nvidia GPUs to accelerate all the steps in the data science process. From data exploration through to model development. Recently, with the inclusion of GPU processing support in Spark 3, RAPIDS has released a set of plugins designed to accelerate Spark DataFrame operations with the aid of GPU accelerators. This article gives an introduction to setting up RAPID.ai on Cloudera's CDP Platform. Before starting, ensure that you have access to the following requirements:  Prerequisites Cloudera CDP Private Cloud Base 7.1.3+ Cloudera CDS 3 Centos 7 Epel Repo already added and installed Kernel 3.10 (deployment script was just tested on this version) Check run `uname -r` Full administrator and sudo rights Nvidia Drivers Firstly in order to be able to support GPUs in the OS layer, we need to install the Nvidia Hardware drivers. See: cdp_rapids for ansible helper scripts for ins

Automating Starting Services in Apache NiFi and Applying ParametersAutomate all the things! You can call these commands interactively or script all of them with awesome DevOps tools. @Andre Araujo and @dchaffey can tell you more about that.Enable All NiFi Services on the CanvasBy running this three times, I get any stubborn ones or ones that needed something previously running. This could be put into a loop; check the status before trying again.nifi pg-listnifi pg-statusnifi pg-get-servicesThe NiFi CLI has interactive help available and also some good documentation:NiFi CLI Toolkit Guide /opt/demo/nifi-toolkit-1.12.1/bin/cli.sh nifi pg-enable-services -u http://edge2ai-1.dim.local:8080 --processGroupId root /opt/demo/nifi-toolkit-1.12.1/bin/cli.sh nifi pg-enable-services -u http://edge2ai-1.dim.local:8080 --processGroupId root /opt/demo/nifi-toolkit-1.12.1/bin/cli.sh nifi pg-enable-services -u http://edge2ai-1.dim.local:8080 --processGroupId root  We coul

Ambari Metrics System (AMS) collects, aggregates, and serves Hadoop and system metrics in Ambari-managed clusters. Basically AMS has four components Metrics Monitors, Hadoop Sinks, Metrics Collector, and Grafana. Metrics Monitors on each host in the cluster collect system-level metrics and publish to the Metrics Collector.Hadoop Sinks plug in to Hadoop components to publish Hadoop metrics to the Metrics Collector.The Metrics Collector is a daemon that runs on a specific host in the cluster and receives data from the registered publishers, the Monitors and the Sinks.Grafana is a daemon that runs on a specific host in the cluster and serves pre-built dashboards for visualizing metrics collected in the Metrics Collector. In this article, we will be checking how we can troubleshoot AMS-related issues effectively.There are multiple issues which arises in AMS that leads to different types of discrepancy such as c

Sometimes you need real CDC and you have access to transaction change logs and you use a tool like QLIK REPLICATE or GoldenGate to pump out records to Kafka, and then Flink SQL or NiFi can read them and process them.Other times you need something easier for just some basic changes and inserts to some tables you are interested in receiving new data as events. Apache NiFi can do this easily for you with QueryDatabaseTableRecord. You don't need to know anything, but the database connection information, table name and what field may change. NiFi will query, watch state and give you new records. Nothing is hardcoded, parameterize those values and you have a generic 'Any RDBMS' to 'Any Other Store' data pipeline. We are reading as records, which means each FlowFile in NiFi can have thousands of records that we know all the fields, types and schema related information for. This can be ones that NiFi infers the schema or ones we use to form a Schema Registry like Cloudera's amazing Open Source

 Credits to @mbawa (Mandeep Singh Bawa) who co-built all the assets in this article. Thank you! We (Mandeep and I) engaged on a customer use case where Cloudera Data Engineering (Spark) jobs were triggered once a file lands in S3 (details on how to trigger CDE from Lambda here). Triggering CDE jobs is quite simple; however, we needed much more. Here are a few of the requirements:Decoupling Ingestion Layer / Processing LayerDecoupling apps (sender) from SparkApps can send and forget payloads without the burden of configuring Spark (#number of executors, memory/cpu, etc), the concern of Spark availability (Upgrades, resources availability, etc), or application impacts from CDE API changesReal-time changes to where CDE jobs are sent (Multi CDE)Monitor job status and alertsMonitoring job run times and alerts which may be out-of-spec runtimesFailover to Secondary CDEThrottlingAuthenticationIt may look as though we are trying to make NiFi into an orchestration engine for CDE.

Because spark action in oozie is not supported in HDP 2.3.x and HDP 2.4.0, there is no workaround especially in kerberos environment. We can use either java action or shell action to launch spark job in oozie workflow. In this article, we will discuss how to use oozie shell action to run a spark job in kerberos environment. Prerequisite:1. Spark client is installed on every host where nodemanager is running. This is because we have no control over which node the2. Optionally, if the spark job need to interact with hbase cluster, hbase client need to be installed on every host as well.Steps:1. Create a shell script with the spark-submit command. For example, in the script.sh:/usr/hdp/current/spark-client/bin/spark-submit --keytab keytab --principal ambari-qa-falconJ@FALCONJSECURE.COM --class org.apache.spark.examples.SparkPi --master yarn-client --driver-memory 500m --num-executors 1 --executor-memory 500m --executor-cores 1 spark-examples.jar 32. Prepare kerberos keytab which will be u

To begin you have 2 options.1. Download the Apache NiFi package onto hn0 of your HDI cluster and start nifi using "NIFIHOME"/bin/nifi.sh start2. Install NiFi through AmbariSee this link for a walkthrough of getting the Ambari nifi installer loaded. : https://community.hortonworks.com/content/kbentry/1282/sample-hdfnifi-flow-to-push-tweets-into-solrbanana.htmlTo install Nifi, start the 'Install Wizard': Open Ambari then: On bottom left -> Actions -> Add service -> check NiFi server -> Next -> Next -> Change any config you like (e.g. install dir, port, setup_prebuilt or values in nifi.properties) -> Next -> Deploy. This will kick off the install which will run for 5-10min.That will get you a NiFi instance on your Cluster hn0 and next you will want to set up your laptop to be able to connect to NiFi. To do this you first need to set up an ssh tunnel into hn0. Ex. ssh -L9000:localhost:22 user@example.azurehdinsight.netthis will route all traff

The distributed nature of HBase, coupled with the concepts of an ordered write log and a log-structured merge tree, makes HBase a great database for large scale data processing. Over the years, HBase has proven itself to be a reliable storage mechanism when you need random, realtime read/write access to your Big Data. HBase is able to deliver because of a prioritization mechanism for memory utilization and caching structures over disk I/O. Memory store, implemented as the MemStore, accumulates data edits as they’re received, buffering them in memory (1). The block cache, an implementation of the BlockCache interface, keeps data blocks resident in memory after they’re read.OVERVIEW OF HBASE I/O Memstore The MemStore is important for accessing recent edits. Without the MemStore, accessing that data as it was written into the write log would require reading and deserializing entries back out of that file, at least a O(n)operation. Instead, MemStore maintains a skiplist structure, wh

Custom parcels and CSDs allow partners and clients to bring their libraries and packages to CDP.  Traditionally, customers would use recipes to install parcels and CSDs or manually copy them to appropriate nodes. But, with CDP Public Cloud, you need a modular approach to do this. To address that need, Cloudera has developed the capability of specifying the parcel and CSD information as part of the cluster template (blueprint). With this approach, the parcel and CSD will get downloaded, distributed and activated as part of the cluster build process.In the example below, FLINK 1.12 should be installed by default, but a different version of the FLINK parcel and CSD are passed for the Cloudera Manager to download, distribute and activate. FLINK service is added to the cluster during the cluster import.High level steps are as follows:Modify Cluster Template to define roletype, servicetype, hostgroup configs. Get the Image Catalog IDCreate a Custom Request Template : add Parcels/CS

Starting CDP 7.2.7, Knox can be configured to load-balance backend service instances as well as support sticky sessions in HA mode. Prior to this when HA was configured for Knox (using the HaProvider in Knox topology) Knox would route requests to a specific backend service, except in case of failover where Knox would pick the next backend service from the configured list.This approach had some limitations, Knox would only serve one backend, even when there were multiple backends available. This especially was a problem for stateless services that needed even load distribution. Knox also had no support for sticky sessions. With this release, Knox can be configured to load-balance requests for multiple backend service instances configured with HaProvider. Currently, load-balancing is done in a simple round-robin fashion. Following is an example of how to configure only load-balancing for an example HA service WHOAMI  <provider> <role>ha</role>

Many access control policies require additional context outside of the resources and security principals that are used by default to evaluate policy decisions. For example, knowledge regarding the time of day and the geographic source of attempted access may dictate whether that access is allowed or denied. Ranger policy evaluation occurs in three distinct steps, 1) request creation, 2) policy evaluation, and 3) post-evaluation. In order to extend the Ranger plugin for a particular service, the request context has to be enriched. For example, enriching the authorization request with the time of day or originating geographic location is needed to evaluate those policies effectively. The first step to enriching the request context is to register a Context Enricher with the service in question. For this example, we will use Hive and our goal will be to enrich the context with its geographic origination. Ranger 0.6 includes the RangerFileBasedGeolocationProvider, which can be used to add