Resources

Cloudera Machine Learning provides a number of methods of connecting to other CDP services and experiences such as a Cloudera Data Warehouse. In this post, we will connect using Python and the Impyla library, as well as using the embedded Cloudera Data Visualization. Using Impyla  Within Cloudera Machine Learning, create a new project and set the language to Python 3.6. The connection details are available from the Data Warehouse console by copying the JDBC connection details which will look like.jdbc:impala://coordinator-aws-2-impala-prod.env-j2ln9x.dw.ylcu-atmi.cloudera.site:443/default;AuthMech=3;transportMode=http;httpPath=cliservice;ssl=1;UID=<workload username>;PWD=<workload password> Use the following Python code to install Impyla and configure a connection:!pip3 install impyla==0.16a3USERNAME='<workload username>'IMPALA_HOST='coordinator-aws-2-impala-prod.env-j2ln9x.dw.ylcu-atmi.cloudera.site'IMPALA_PORT='443'from impala.dbapi import connectconn =

Here is a fun one: how do you connect from Python in Cloudera Machine Learning to our Kafka datahub cluster?The documentation is pretty thorough, but it does not have an example of a python client. That's what I'm going to highlight in this article. The good news is that since CML and Datahub run in the same network, you don't need to worry about opening the broker ports, therefore you just need to follow these steps:Step 1: Get and upload your freeIPA certificateStep 2: Find your broker hostnamesStep 3: Setup your clientStep 1: Get and upload your freeIPA certificateGo to your management console > your environment > Actions > Get FreeIPA Certificate:Once downloaded, go to your CML workspace, and upload your file (e.g. /home/cdsw/ca.crt).Step 2: Find your broker hostnamesFor this, go to your Kafka Datahub Cluster > CM UI > Kafka > Instances; you can find the broker hosts here:Step 3: Setup your clientThen, open a session in CML, and use the following paramet

Cloudera Machine Learning provides support for Python3. It is very straightforward to connect a session with an operational database. Provision an Operational Database Log into a CDP instance Select Operational Database Select Create Database Choose the Cloud environment Provide a unique name for the database Click Create Database Once the database has started, make a copy of the Phoenix (Thin) JDBC URL. This will be used as the connection string. Create a Machine Learning Project Within your Cloudera Machine Learning (CML) workspace, create a new project. Provide a name, and choose a blank initial setup. Create a session, and install phoenixdb using the command:!pip3 install phoenixdb Create a new Python file and paste the following code into the notebook.Import the required dependenciesimport phoenixdbimport ioimport json Setup the parameters required to establish the connection with ODB. Refer to the Thin client details. opts = {}opts['authentication'] = 'BASIC'op

In order to convert JSON to XML, you can use ConvertRecord processor available as part of the NiFi tool (I worked on version: 1.11.4). Following are the steps to configure ConvertRecord processor to convert JSON input to XML output: Add a processor – ConvertRecord and right-click to “configure”  Click the Record Reader value and select Create New Service. Select JSONTreeReader and click CREATE.  Similarly, click on the Record Writer value and select Create New Service. Select XMLRecordSetWriter and click CREATE. Final configuration of ConvertRecord should look like this: Click à to configure JSONTreeReader to configure details about the schema that should be read by the registry. The controller service form should open: Click configure of JSONTreeReader to start configuring the JSON Reader and configure as per the following screenshot: When you select Create New Service in the above screenshot, another form opens up. Fill it up ba

Introduction When running Apache Kafka at scale, sooner or later you will start worrying about available disk space, growth rates of topics, or general disk usage. This is especially the case when handling with external producers - outside of your direct control and without configuring hard size limits within your retention configuration. The kafka-log-dirs tool allows you to determine the current size and location of all available partitions on your Kafka cluster. Due to its nature, this tool only represents a snapshot of the current state without any history - and it is far from being convenient to use. You want to know how far? Here's a snippet of the output:   But we want to turn the JSON above into something more shiny and usable, like this:   So, let us figure out how we can facilitate Apache NiFi, Apache Kudu, and Tableau to retrieve consumable visualization and actionable insights about your Kafka cluster! Architecture On a high level, we are going to use Apache NiF

This video provides instructions on how to SSH keys in CDP Public Cloud data hub clusters.It touches on how to generate a new SSH key pair and steps through the process of deploying it for a workload user through the Cloudera Management Console Web UI, as well as using the CDP command-line tool. It discusses the security implications of using the Cloudbreak user for login purposes on data hub hosts and explains why workload user credentials should be used instead in most cases. It also demonstrates using the deployed SSH keys for login to data hub hosts.

Recently, the weekly number of COVID-19 cases in The Netherlands has been steadily dropping week over week. However, underneath this lies a hidden positive trend of cases with the British COVID variant.  In this article, I explain how I made my first visual in Cloudera Data Visualization with just a few clicks. Step 1: The Data The data is created by combining several sources that shed light on the percentage of new infections that are made up with this variant, as well as the total number of cases. As we are here for tech, more than science, I will only highlight the official report of cases per week and an official article that contains the latest percentage, as well as an earlier percentage. Additional explanation on the numbers is given in the post below this article.   From here I could have chosen from many data sources, including a CSV file, but for reproducibility, I decided to upload it to Hive with a simple query:   create table covid_cases asSELECT 431 as b

RAPIDS.ai is a set of Nvidia libraries designed to help data scientists and engineers to leverage the Nvidia GPUs to accelerate all the steps in the data science process. From data exploration through to model development. Recently, with the inclusion of GPU processing support in Spark 3, RAPIDS has released a set of plugins designed to accelerate Spark DataFrame operations with the aid of GPU accelerators. This article gives an introduction to setting up RAPID.ai on Cloudera's CDP Platform. Before starting, ensure that you have access to the following requirements:  Prerequisites Cloudera CDP Private Cloud Base 7.1.3+ Cloudera CDS 3 Centos 7 Epel Repo already added and installed Kernel 3.10 (deployment script was just tested on this version) Check run `uname -r` Full administrator and sudo rights Nvidia Drivers Firstly in order to be able to support GPUs in the OS layer, we need to install the Nvidia Hardware drivers. See: cdp_rapids for ansible helper scripts for ins

Automating Starting Services in Apache NiFi and Applying ParametersAutomate all the things! You can call these commands interactively or script all of them with awesome DevOps tools. @Andre Araujo and @dchaffey can tell you more about that.Enable All NiFi Services on the CanvasBy running this three times, I get any stubborn ones or ones that needed something previously running. This could be put into a loop; check the status before trying again.nifi pg-listnifi pg-statusnifi pg-get-servicesThe NiFi CLI has interactive help available and also some good documentation:NiFi CLI Toolkit Guide /opt/demo/nifi-toolkit-1.12.1/bin/cli.sh nifi pg-enable-services -u http://edge2ai-1.dim.local:8080 --processGroupId root /opt/demo/nifi-toolkit-1.12.1/bin/cli.sh nifi pg-enable-services -u http://edge2ai-1.dim.local:8080 --processGroupId root /opt/demo/nifi-toolkit-1.12.1/bin/cli.sh nifi pg-enable-services -u http://edge2ai-1.dim.local:8080 --processGroupId root  We coul

Ambari Metrics System (AMS) collects, aggregates, and serves Hadoop and system metrics in Ambari-managed clusters. Basically AMS has four components Metrics Monitors, Hadoop Sinks, Metrics Collector, and Grafana. Metrics Monitors on each host in the cluster collect system-level metrics and publish to the Metrics Collector.Hadoop Sinks plug in to Hadoop components to publish Hadoop metrics to the Metrics Collector.The Metrics Collector is a daemon that runs on a specific host in the cluster and receives data from the registered publishers, the Monitors and the Sinks.Grafana is a daemon that runs on a specific host in the cluster and serves pre-built dashboards for visualizing metrics collected in the Metrics Collector. In this article, we will be checking how we can troubleshoot AMS-related issues effectively.There are multiple issues which arises in AMS that leads to different types of discrepancy such as c

Sometimes you need real CDC and you have access to transaction change logs and you use a tool like QLIK REPLICATE or GoldenGate to pump out records to Kafka, and then Flink SQL or NiFi can read them and process them.Other times you need something easier for just some basic changes and inserts to some tables you are interested in receiving new data as events. Apache NiFi can do this easily for you with QueryDatabaseTableRecord. You don't need to know anything, but the database connection information, table name and what field may change. NiFi will query, watch state and give you new records. Nothing is hardcoded, parameterize those values and you have a generic 'Any RDBMS' to 'Any Other Store' data pipeline. We are reading as records, which means each FlowFile in NiFi can have thousands of records that we know all the fields, types and schema related information for. This can be ones that NiFi infers the schema or ones we use to form a Schema Registry like Cloudera's amazing Open Source

 Credits to @mbawa (Mandeep Singh Bawa) who co-built all the assets in this article. Thank you! We (Mandeep and I) engaged on a customer use case where Cloudera Data Engineering (Spark) jobs were triggered once a file lands in S3 (details on how to trigger CDE from Lambda here). Triggering CDE jobs is quite simple; however, we needed much more. Here are a few of the requirements:Decoupling Ingestion Layer / Processing LayerDecoupling apps (sender) from SparkApps can send and forget payloads without the burden of configuring Spark (#number of executors, memory/cpu, etc), the concern of Spark availability (Upgrades, resources availability, etc), or application impacts from CDE API changesReal-time changes to where CDE jobs are sent (Multi CDE)Monitor job status and alertsMonitoring job run times and alerts which may be out-of-spec runtimesFailover to Secondary CDEThrottlingAuthenticationIt may look as though we are trying to make NiFi into an orchestration engine for CDE.

Because spark action in oozie is not supported in HDP 2.3.x and HDP 2.4.0, there is no workaround especially in kerberos environment. We can use either java action or shell action to launch spark job in oozie workflow. In this article, we will discuss how to use oozie shell action to run a spark job in kerberos environment. Prerequisite:1. Spark client is installed on every host where nodemanager is running. This is because we have no control over which node the2. Optionally, if the spark job need to interact with hbase cluster, hbase client need to be installed on every host as well.Steps:1. Create a shell script with the spark-submit command. For example, in the script.sh:/usr/hdp/current/spark-client/bin/spark-submit --keytab keytab --principal ambari-qa-falconJ@FALCONJSECURE.COM --class org.apache.spark.examples.SparkPi --master yarn-client --driver-memory 500m --num-executors 1 --executor-memory 500m --executor-cores 1 spark-examples.jar 32. Prepare kerberos keytab which will be u

To begin you have 2 options.1. Download the Apache NiFi package onto hn0 of your HDI cluster and start nifi using "NIFIHOME"/bin/nifi.sh start2. Install NiFi through AmbariSee this link for a walkthrough of getting the Ambari nifi installer loaded. : https://community.hortonworks.com/content/kbentry/1282/sample-hdfnifi-flow-to-push-tweets-into-solrbanana.htmlTo install Nifi, start the 'Install Wizard': Open Ambari then: On bottom left -> Actions -> Add service -> check NiFi server -> Next -> Next -> Change any config you like (e.g. install dir, port, setup_prebuilt or values in nifi.properties) -> Next -> Deploy. This will kick off the install which will run for 5-10min.That will get you a NiFi instance on your Cluster hn0 and next you will want to set up your laptop to be able to connect to NiFi. To do this you first need to set up an ssh tunnel into hn0. Ex. ssh -L9000:localhost:22 user@example.azurehdinsight.netthis will route all traff

The distributed nature of HBase, coupled with the concepts of an ordered write log and a log-structured merge tree, makes HBase a great database for large scale data processing. Over the years, HBase has proven itself to be a reliable storage mechanism when you need random, realtime read/write access to your Big Data. HBase is able to deliver because of a prioritization mechanism for memory utilization and caching structures over disk I/O. Memory store, implemented as the MemStore, accumulates data edits as they’re received, buffering them in memory (1). The block cache, an implementation of the BlockCache interface, keeps data blocks resident in memory after they’re read.OVERVIEW OF HBASE I/O Memstore The MemStore is important for accessing recent edits. Without the MemStore, accessing that data as it was written into the write log would require reading and deserializing entries back out of that file, at least a O(n)operation. Instead, MemStore maintains a skiplist structure, wh

Custom parcels and CSDs allow partners and clients to bring their libraries and packages to CDP.  Traditionally, customers would use recipes to install parcels and CSDs or manually copy them to appropriate nodes. But, with CDP Public Cloud, you need a modular approach to do this. To address that need, Cloudera has developed the capability of specifying the parcel and CSD information as part of the cluster template (blueprint). With this approach, the parcel and CSD will get downloaded, distributed and activated as part of the cluster build process.In the example below, FLINK 1.12 should be installed by default, but a different version of the FLINK parcel and CSD are passed for the Cloudera Manager to download, distribute and activate. FLINK service is added to the cluster during the cluster import.High level steps are as follows:Modify Cluster Template to define roletype, servicetype, hostgroup configs. Get the Image Catalog IDCreate a Custom Request Template : add Parcels/CS

Starting CDP 7.2.7, Knox can be configured to load-balance backend service instances as well as support sticky sessions in HA mode. Prior to this when HA was configured for Knox (using the HaProvider in Knox topology) Knox would route requests to a specific backend service, except in case of failover where Knox would pick the next backend service from the configured list.This approach had some limitations, Knox would only serve one backend, even when there were multiple backends available. This especially was a problem for stateless services that needed even load distribution. Knox also had no support for sticky sessions. With this release, Knox can be configured to load-balance requests for multiple backend service instances configured with HaProvider. Currently, load-balancing is done in a simple round-robin fashion. Following is an example of how to configure only load-balancing for an example HA service WHOAMI  <provider> <role>ha</role>

Many access control policies require additional context outside of the resources and security principals that are used by default to evaluate policy decisions. For example, knowledge regarding the time of day and the geographic source of attempted access may dictate whether that access is allowed or denied. Ranger policy evaluation occurs in three distinct steps, 1) request creation, 2) policy evaluation, and 3) post-evaluation. In order to extend the Ranger plugin for a particular service, the request context has to be enriched. For example, enriching the authorization request with the time of day or originating geographic location is needed to evaluate those policies effectively. The first step to enriching the request context is to register a Context Enricher with the service in question. For this example, we will use Hive and our goal will be to enrich the context with its geographic origination. Ranger 0.6 includes the RangerFileBasedGeolocationProvider, which can be used to add

In part 1 of this series we talked about the growing relevance of streaming technologies and covered the need for existing Cloudera customers currently using Apache Flume, to consider moving over to Cloudera DataFlow (CDF).   Cloudera DataFlow is an umbrella term that covers the streaming technologies from Cloudera. CDF is supported on CDH 5 / CDH 6 and HDP 2 / HDP 3. So there is nothing stopping customers adopting Cloudera DataFlow right now so that they are in a supported configuration for when they upgrade to the new Cloudera Data Platform (CDP).   CDF includes the technology to address a number of areas: Edge Flow Management Core Flow Management Stream Processing Streaming Analytics Enterprises Services A good summary of these components can be found in the blog post Introducing Cloudera Data Flow.   Cloudera DataFlow - Data-In-Motion Platform   If you are a traditional Cloudera customer using the Cloudera Distribution of Apache Kafka, there are a number of

Cloudera Data Platform recently introduced Cluster Connectivity Manager (CCM) in tech preview, a feature with which CDP can communicate with workload clusters without the need for inbound network connections to customer workload clusters.   In this article, I will highlight how to automate the setup of CCM for both AWS and Azure including: Cloud Network Setup CDP Environment setup   Before you continue reading: You can find all automation on my github (here) Please refer to the official documentation for more details (here) This is just an example, different deployments require a different set of parameters! Overview   As highlighted in the above image, CCM uses reverse tunneling to communicate back to your CDP dedicated control plane. A few things to note: Each tenant (i.e. customer) will have their own dedicated tunnel using a dedicated Apache Mina server The diagram above describes a setup in AWS. The Azure setup is very similar but does not require public sub

COD - CDE using PhoenixIn this article, we will walk through steps required to be followed to read/ write using Phoenix JDBC to COD (Cloudera Operational Database) from Spark on CDE (Cloudera Data Engineering).AssumptionCOD is already provisioned and database cod-db is created. Refer to this link for the same.CDE is already provisioned and virtual cluster is already created. Refer to this link for the same.CODConfigurationSpark in CDE to be able to talk to COD, it would require the hbase-site.xml config of the COD cluster. Do the following steps for retrieve the same:Go to the COD control plane UI and click on cod-db database.Under the "Connect" tab of COD database, look for HBase Client Configuration URL field. Following is the screenshot for the same.The configuration can be downloaded using the following curl command. curl -o "hbase-config.zip" -u "santhosh" "https://XXXXXX.cldr.work/clouderamanager/api/v41/clusters/cod-hvtur2ovawfr/services/hbase/

In this article, we’ll walk through the steps required to connect a Spark Structured Streaming application to Kafka in CDP Data Engineering Experience (DEX). This article extends the official CDP documentation: Connecting Kafka clients to Data Hub provisioned clusters, to include Spark applications run in Cloudera Data Experience.   Steps Obtain the FreeIPA certificate of your environment: From the CDP home page, navigate to Management Console > Environments Locate and select your environment from the list of available environments Click Actions Select Get FreeIPA Certificate from the drop-down menu. The FreeIPA certificate downloads. Add the FreeIPA certificate to a Java keystore: All clients that you want to connect to the Data Hub provisioned cluster, will need to use the certificate to communicate over TLS. The exact steps of adding the certificate to the truststore depends on the platform and key management software used. For example

In this article, we will walk through the steps required to connect a Spark Structured Streaming application to Kafka in CDP Data Hub. We use two Data Hubs, one with a Data Engineering Template, and another with a Streams Messaging template. Both Data Hubs were created in the same environment. This article extends the official CDP documentation: Connecting Kafka clients to Data Hub provisioned clusters to include Spark applications. Steps1. Obtain the FreeIPA certificate of your environment:From the CDP Home Page, navigate to Management Console > EnvironmentsLocate and select your environment from the list of available environmentsClick ActionsSelect Get FreeIPA Certificate from the drop-down menu. The FreeIPA certificate downloads.2. Add the FreeIPA certificate to the truststore of the client.The certificate needs to be added for all clients that you want to connect to the Data Hub provisioned cluster. The exact steps of adding the certi

 Sometimes you need real CDC and you have access to transaction change logs and you use a tool like QLIK REPLICATE or GoldenGate to pump out records to Kafka and then Flink SQL or NiFi can read them and process them.Other times you need something easier for just some basic changes and inserts to some tables you are interested in receiving new data as events. Apache NiFi can do this easily for you with QueryDatabaseTableRecord. you don't need to know anything but the database connection information, table name and what field may change. NiFi will query, watch state and give you new records. Nothing is hardcoded, parameterize those values and you have a generic Any RDBMS to Any Other Store data pipeline. We are reading as records, which means each FlowFile in NiFi can have thousands of records that we know all the fields, types and schema related information for. This can be ones that NiFi infers the schema or ones we use from a Schema Registry like Cloudera's amazing Open Source Sc

OverviewCDP Public Cloud provides a mechanism for provisioning all of your resources in private subnets (i.e. resources that don't get assigned public IPs). I won't detail the specifics about how each CSP accomplishes this; you can find much more thorough overviews in each CSP's documentation.However, in using this architecture, the problem becomes: how do I interact (UI/API/etc) with these resources if they aren't publicly accessible? In an ideal/ corporate world, some sort of VPN peering would be setup so that merely being on your corporate VPN would allow access to the resources in these subnets.  You may find yourself in the unfortunate situation I do, where, you happen to be using an AWS (or Azure) account that doesn't have the fancy VPN peering enabled. Enter the SOCKS5 proxy. I also won't go over the specifics about how this works, but suffice it to say, we will create a bastion host in a public subnet (with a very narrow security group configuration), create an S